You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Marco Matarazzo <ma...@hexkeep.com> on 2013/02/17 12:51:21 UTC
nodetool repair with vnodes
Greetings.
I'm trying to run "nodetool repair" on a Cassandra 1.2.1 cluster of 3 nodes with 256 vnodes each.
On a pre-1.2 cluster I used to launch a "nodetool repair" on every node every 24hrs. Now I'm getting a differenf behavior, and I'm sure I'm missing something.
What I see on the command line is:
[2013-02-17 10:20:15,186] Starting repair command #1, repairing 768 ranges for keyspace goh_master
[2013-02-17 10:48:13,401] Repair session 3d140e10-78e3-11e2-af53-d344dbdd69f5 for range (6556914650761469337,6580337080281832001] finished
(…repeat the last line 767 times)
…so it seems to me that it is running on all vnodes ranges.
Also, whatever the node which I launch the command on is, only one node log is "moving" and is always the same node.
So, to me, it's like the "nodetool repair" command is running always on the same single node and repairing everything.
I'm sure I'm making some mistakes, and I just can't find any clue of what's wrong with my nodetool usage on the documentation (if anything is wrong, btw). Is there anything I'm missing ?
--
Marco Matarazzo
Re: nodetool repair with vnodes
Posted by aaron morton <aa...@thelastpickle.com>.
> So, running it periodically on just one node is enough for cluster maintenance ?
In the special case where you have RF == Number of nodes.
The recommended approach is to use -pr and run it on each node periodically.
> Also: running it with -pr does output:
That does not look right. There should be messages about requesting and receiving merkle tree's from other nodes, and that certain CF's are in sync. These are all logged from the AntiEntropyService.
> Is there a way to run it only for all vnodes on a single physical node ?
it should be doing that.
Look for messages like this in the log
logger.info(String.format("[repair #%s] new session: will sync %s on range %s for %s.%s", getName(), repairedNodes(), range, tablename, Arrays.toString(cfnames)));
They say how much is going to be synced, and with what. Try running repair with -pr on one of nodes not already repaired.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 18/02/2013, at 11:12 AM, Marco Matarazzo <ma...@hexkeep.com> wrote:
>>> So, to me, it's like the "nodetool repair" command is running always on the same single node and repairing everything.
>> If you use nodetool repair without the -pr flag in your setup (3 nodes and I assume RF 3) it will repair all token ranges in the cluster.
>
> That's correct, 3 nodes and RF 3. Sorry for not specifying it in the beginning.
>
>
> So, running it periodically on just one node is enough for cluster maintenance ? Does this depends on the fact that every vnode data is related with the previous and next vnode, and this particular setup makes this enough as it cover every physical node?
>
>
> Also: running it with -pr does output:
>
> [2013-02-17 12:29:25,293] Nothing to repair for keyspace 'system'
> [2013-02-17 12:29:25,301] Starting repair command #2, repairing 1 ranges for keyspace keyspace_test
> [2013-02-17 12:29:28,028] Repair session 487d0650-78f5-11e2-a73a-2f5b109ee83c for range (-9177680845984855691,-9171525326632276709] finished
> [2013-02-17 12:29:28,028] Repair command #2 finished
>
> … that, as far as I can understand, works on the first vnode on the specified node, or so it seems from the output range. Am I right? Is there a way to run it only for all vnodes on a single physical node ?
>
> Thank you!
>
> --
> Marco Matarazzo
Re: nodetool repair with vnodes
Posted by Marco Matarazzo <ma...@hexkeep.com>.
>> So, to me, it's like the "nodetool repair" command is running always on the same single node and repairing everything.
> If you use nodetool repair without the -pr flag in your setup (3 nodes and I assume RF 3) it will repair all token ranges in the cluster.
That's correct, 3 nodes and RF 3. Sorry for not specifying it in the beginning.
So, running it periodically on just one node is enough for cluster maintenance ? Does this depends on the fact that every vnode data is related with the previous and next vnode, and this particular setup makes this enough as it cover every physical node?
Also: running it with -pr does output:
[2013-02-17 12:29:25,293] Nothing to repair for keyspace 'system'
[2013-02-17 12:29:25,301] Starting repair command #2, repairing 1 ranges for keyspace keyspace_test
[2013-02-17 12:29:28,028] Repair session 487d0650-78f5-11e2-a73a-2f5b109ee83c for range (-9177680845984855691,-9171525326632276709] finished
[2013-02-17 12:29:28,028] Repair command #2 finished
… that, as far as I can understand, works on the first vnode on the specified node, or so it seems from the output range. Am I right? Is there a way to run it only for all vnodes on a single physical node ?
Thank you!
--
Marco Matarazzo
Re: nodetool repair with vnodes
Posted by aaron morton <aa...@thelastpickle.com>.
> …so it seems to me that it is running on all vnodes ranges.
Yes.
> Also, whatever the node which I launch the command on is, only one node log is "moving" and is always the same node.
Not sure what you mean here.
> So, to me, it's like the "nodetool repair" command is running always on the same single node and repairing everything.
If you use nodetool repair without the -pr flag in your setup (3 nodes and I assume RF 3) it will repair all token ranges in the cluster.
> Is there anything I'm missing ?
Look for messages with "session completed" in the log from the AntiEntropyService
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 18/02/2013, at 12:51 AM, Marco Matarazzo <ma...@hexkeep.com> wrote:
> Greetings.
>
> I'm trying to run "nodetool repair" on a Cassandra 1.2.1 cluster of 3 nodes with 256 vnodes each.
>
> On a pre-1.2 cluster I used to launch a "nodetool repair" on every node every 24hrs. Now I'm getting a differenf behavior, and I'm sure I'm missing something.
>
> What I see on the command line is:
>
> [2013-02-17 10:20:15,186] Starting repair command #1, repairing 768 ranges for keyspace goh_master
> [2013-02-17 10:48:13,401] Repair session 3d140e10-78e3-11e2-af53-d344dbdd69f5 for range (6556914650761469337,6580337080281832001] finished
> (…repeat the last line 767 times)
>
> …so it seems to me that it is running on all vnodes ranges.
>
> Also, whatever the node which I launch the command on is, only one node log is "moving" and is always the same node.
>
> So, to me, it's like the "nodetool repair" command is running always on the same single node and repairing everything.
>
> I'm sure I'm making some mistakes, and I just can't find any clue of what's wrong with my nodetool usage on the documentation (if anything is wrong, btw). Is there anything I'm missing ?
>
> --
> Marco Matarazzo
>
>