You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Artur Siekielski <ar...@vhex.net> on 2016/11/09 20:22:50 UTC
repair -pr in crontab
Hi,
the docs give me an impression that repairing should be run manually,
and not put in crontab for default. Should each repair run be monitored
manually?
If I would like to put "repair -pr" in crontab for each node, with a few
hour difference between the runs, are there any risks with such setup?
Specifically:
- if two or more "repair -pr" runs on different nodes are running at the
same time, can it cause any problems besides high load?
- can "repair -pr" be run simultaneously on all nodes at the same time?
- I'm using the default gc_grace_period of 10 days. Are there any
reasons to run repairing more often that once per 10 days, for a case
when previous repairing fails?
- how to monitor start and finish times of repairs, and if the runs were
successful? Does the "nodetool repair" command is guaranteed to exit
only after the repair is finished and does it return a status code to a
shell?
Re: repair -pr in crontab
Posted by Benjamin Roth <be...@jaumo.com>.
It is absolutely ok to run parallel repair -pr, if
1. the ranges do not overlap
2. if your cluster can handle the pressure - do not underestimate that.
In reaper you can tweak some settings like repair intensity to give your
cluster some time to breath between repair slices.
2016-11-25 11:34 GMT+01:00 Artur Siekielski <ar...@vhex.net>:
> Hi,
> yes, I read about how the repairing works, but the docs/blog posts lack
> practical recommendations and "best practices". For example, I found people
> having issues with running "repair -pr" simultaneously on all nodes, but it
> isn't clear it shouldn't be allowed.
>
> In the end I implemented rolling, sequential "repair -pr" run on all nodes
> (it's pretty easy to implement when you have Salt/Ansible, or even ssh).
>
>
> On 11/25/2016 07:30 AM, wxn002@zjqunshuo.com wrote:
>
>> Hi Artur,
>> When I asked similar questions, someone addressed me to the below links
>> and they are helpful.
>>
>> See http://www.datastax.com/dev/blog/repair-in-cassandra
>> https://lostechies.com/ryansvihla/2015/09/25/cassandras-
>> repair-should-be-called-required-maintenance/
>> https://cassandra-zone.com/understanding-repairs/
>>
>> Cheers,
>> -Simon
>>
>>
>
--
Benjamin Roth
Prokurist
Jaumo GmbH · www.jaumo.com
Wehrstraße 46 · 73035 Göppingen · Germany
Phone +49 7161 304880-6 · Fax +49 7161 304880-1
AG Ulm · HRB 731058 · Managing Director: Jens Kammerer
Re: repair -pr in crontab
Posted by Artur Siekielski <ar...@vhex.net>.
Hi,
yes, I read about how the repairing works, but the docs/blog posts lack
practical recommendations and "best practices". For example, I found
people having issues with running "repair -pr" simultaneously on all
nodes, but it isn't clear it shouldn't be allowed.
In the end I implemented rolling, sequential "repair -pr" run on all
nodes (it's pretty easy to implement when you have Salt/Ansible, or even
ssh).
On 11/25/2016 07:30 AM, wxn002@zjqunshuo.com wrote:
> Hi Artur,
> When I asked similar questions, someone addressed me to the below links
> and they are helpful.
>
> See http://www.datastax.com/dev/blog/repair-in-cassandra
> https://lostechies.com/ryansvihla/2015/09/25/cassandras-repair-should-be-called-required-maintenance/
> https://cassandra-zone.com/understanding-repairs/
>
> Cheers,
> -Simon
>
Re: repair -pr in crontab
Posted by Alexander Dejanovski <al...@thelastpickle.com>.
Hi,
we maintain a hard fork of Reaper that works with all versions of Cassandra
up to 3.0.x : https://github.com/thelastpickle/cassandra-reaper
Just to save you some time digging into all the forks that could exist.
Cheers,
On Fri, Nov 25, 2016 at 7:37 AM Benjamin Roth <be...@jaumo.com>
wrote:
> I recommend using cassandra-reaper
> Using crons without proper Monitoring will most likely not work as
> expected.
> There are some reaper forks on GitHub. You have to check which one works
> with your Cassandra version. The original one from Spotify only works on
> 2.x not on 3.x
>
> Am 25.11.2016 07:31 schrieb "wxn002@zjqunshuo.com" <wx...@zjqunshuo.com>:
>
> Hi Artur,
> When I asked similar questions, someone addressed me to the below links
> and they are helpful.
>
> See http://www.datastax.com/dev/blog/repair-in-cassandra
>
> https://lostechies.com/ryansvihla/2015/09/25/cassandras-repair-should-be-called-required-maintenance/
> https://cassandra-zone.com/understanding-repairs/
>
> Cheers,
> -Simon
>
> *From:* Artur Siekielski <ar...@vhex.net>
> *Date:* 2016-11-10 04:22
> *To:* user <us...@cassandra.apache.org>
> *Subject:* repair -pr in crontab
> Hi,
> the docs give me an impression that repairing should be run manually,
> and not put in crontab for default. Should each repair run be monitored
> manually?
>
> If I would like to put "repair -pr" in crontab for each node, with a few
> hour difference between the runs, are there any risks with such setup?
> Specifically:
> - if two or more "repair -pr" runs on different nodes are running at the
> same time, can it cause any problems besides high load?
> - can "repair -pr" be run simultaneously on all nodes at the same time?
> - I'm using the default gc_grace_period of 10 days. Are there any
> reasons to run repairing more often that once per 10 days, for a case
> when previous repairing fails?
> - how to monitor start and finish times of repairs, and if the runs were
> successful? Does the "nodetool repair" command is guaranteed to exit
> only after the repair is finished and does it return a status code to a
> shell?
>
> --
-----------------
Alexander Dejanovski
France
@alexanderdeja
Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
Re: repair -pr in crontab
Posted by Benjamin Roth <be...@jaumo.com>.
I recommend using cassandra-reaper
Using crons without proper Monitoring will most likely not work as
expected.
There are some reaper forks on GitHub. You have to check which one works
with your Cassandra version. The original one from Spotify only works on
2.x not on 3.x
Am 25.11.2016 07:31 schrieb "wxn002@zjqunshuo.com" <wx...@zjqunshuo.com>:
> Hi Artur,
> When I asked similar questions, someone addressed me to the below links
> and they are helpful.
>
> See http://www.datastax.com/dev/blog/repair-in-cassandra
> https://lostechies.com/ryansvihla/2015/09/25/cassandras-repair-should-be-
> called-required-maintenance/
> https://cassandra-zone.com/understanding-repairs/
>
> Cheers,
> -Simon
>
> *From:* Artur Siekielski <ar...@vhex.net>
> *Date:* 2016-11-10 04:22
> *To:* user <us...@cassandra.apache.org>
> *Subject:* repair -pr in crontab
> Hi,
> the docs give me an impression that repairing should be run manually,
> and not put in crontab for default. Should each repair run be monitored
> manually?
>
> If I would like to put "repair -pr" in crontab for each node, with a few
> hour difference between the runs, are there any risks with such setup?
> Specifically:
> - if two or more "repair -pr" runs on different nodes are running at the
> same time, can it cause any problems besides high load?
> - can "repair -pr" be run simultaneously on all nodes at the same time?
> - I'm using the default gc_grace_period of 10 days. Are there any
> reasons to run repairing more often that once per 10 days, for a case
> when previous repairing fails?
> - how to monitor start and finish times of repairs, and if the runs were
> successful? Does the "nodetool repair" command is guaranteed to exit
> only after the repair is finished and does it return a status code to a
> shell?
>
>
Re: repair -pr in crontab
Posted by "wxn002@zjqunshuo.com" <wx...@zjqunshuo.com>.
Hi Artur,
When I asked similar questions, someone addressed me to the below links and they are helpful.
See http://www.datastax.com/dev/blog/repair-in-cassandra
https://lostechies.com/ryansvihla/2015/09/25/cassandras-repair-should-be-called-required-maintenance/
https://cassandra-zone.com/understanding-repairs/
Cheers,
-Simon
From: Artur Siekielski
Date: 2016-11-10 04:22
To: user
Subject: repair -pr in crontab
Hi,
the docs give me an impression that repairing should be run manually,
and not put in crontab for default. Should each repair run be monitored
manually?
If I would like to put "repair -pr" in crontab for each node, with a few
hour difference between the runs, are there any risks with such setup?
Specifically:
- if two or more "repair -pr" runs on different nodes are running at the
same time, can it cause any problems besides high load?
- can "repair -pr" be run simultaneously on all nodes at the same time?
- I'm using the default gc_grace_period of 10 days. Are there any
reasons to run repairing more often that once per 10 days, for a case
when previous repairing fails?
- how to monitor start and finish times of repairs, and if the runs were
successful? Does the "nodetool repair" command is guaranteed to exit
only after the repair is finished and does it return a status code to a
shell?