You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Kevin Burton <bu...@spinn3r.com> on 2014/08/21 21:32:05 UTC
stalled nodetool repair?
How do I watch the progress of nodetool repair.
Looks like the folklore from the list says to just use
nodetool compactionstats
nodetool netstats
… but the repair seems locked/stalled and neither of these are showing any
progress..
granted , this is a lot of data, but it would be nice to at least see some
progress.
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: stalled nodetool repair?
Posted by Aiman Parvaiz <ai...@grapheffect.com>.
If nodetool compactionstats says there are no Validation compactions
running (and the compaction queue is empty) and netstats says there is
nothing streaming there is a a good chance the repair is finished or dead.
Source:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Is-it-safe-to-stop-a-read-repair-and-any-suggestion-on-speeding-up-repairs-td6607367.html
You might find this helpful.
Thanks
On Thu, Aug 21, 2014 at 12:32 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> How do I watch the progress of nodetool repair.
>
> Looks like the folklore from the list says to just use
>
> nodetool compactionstats
> nodetool netstats
>
> … but the repair seems locked/stalled and neither of these are showing any
> progress..
>
> granted , this is a lot of data, but it would be nice to at least see some
> progress.
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>
Re: stalled nodetool repair?
Posted by DuyHai Doan <do...@gmail.com>.
Great! Many thanks
On Thu, Aug 21, 2014 at 9:35 PM, Ben Bromhead <be...@instaclustr.com> wrote:
> Ah sorry that is the original repo, see
> https://github.com/BrianGallew/cassandra_range_repair for the updated
> version of the script with vnode support
>
> Ben Bromhead
> Instaclustr | www.instaclustr.com | @instaclustr
> <http://twitter.com/instaclustr> | +61 415 936 359
>
> On 22 Aug 2014, at 2:19 pm, DuyHai Doan <do...@gmail.com> wrote:
>
> Thanks Ben for the link. Still this script does not work with vnodes,
> which exclude a wide range of C* config
>
>
> On Thu, Aug 21, 2014 at 5:51 PM, Ben Bromhead <be...@instaclustr.com> wrote:
>
>> https://github.com/mstump/cassandra_range_repair
>>
>> Also very useful.
>>
>> Ben Bromhead
>> Instaclustr | www.instaclustr.com | @instaclustr
>> <http://twitter.com/instaclustr> | +61 415 936 359
>>
>>
>>
>>
>> On 22/08/2014, at 6:12 AM, Robert Coli <rc...@eventbrite.com> wrote:
>>
>> On Thu, Aug 21, 2014 at 12:32 PM, Kevin Burton <bu...@spinn3r.com>
>> wrote:
>>
>>> How do I watch the progress of nodetool repair.
>>>
>>
>> This is a very longstanding operational problem in Cassandra. Repair
>> barely works and is opaque, yet one is expected to run it once a week in
>> the default configuration.
>>
>> An unreasonably-hostile-in-tone-but-otherwise-accurate description of the
>> status quo before the re-write of streaming in 2.0 :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-5396
>>
>> A proposal to change the default for gc_grace_seconds to 34 days, so that
>> this fragile and heavyweight operation only has to be done once a month :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-5850
>>
>>
>>> granted , this is a lot of data, but it would be nice to at least see
>>> some progress.
>>>
>>
>> Here's the rewrite of streaming, where progress indication improves
>> dramatically over the prior status quo :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-5286
>>
>> And here's two open tickets on making repair less opaque (thx yukim@#cassandra)
>> :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-5483
>> https://issues.apache.org/jira/browse/CASSANDRA-5839
>>
>> =Rob
>>
>>
>>
>>
>
>
Re: stalled nodetool repair?
Posted by Ben Bromhead <be...@instaclustr.com>.
Ah sorry that is the original repo, see https://github.com/BrianGallew/cassandra_range_repair for the updated version of the script with vnode support
Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359
On 22 Aug 2014, at 2:19 pm, DuyHai Doan <do...@gmail.com> wrote:
> Thanks Ben for the link. Still this script does not work with vnodes, which exclude a wide range of C* config
>
>
> On Thu, Aug 21, 2014 at 5:51 PM, Ben Bromhead <be...@instaclustr.com> wrote:
> https://github.com/mstump/cassandra_range_repair
>
> Also very useful.
>
> Ben Bromhead
> Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359
>
>
>
>
> On 22/08/2014, at 6:12 AM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Thu, Aug 21, 2014 at 12:32 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>> How do I watch the progress of nodetool repair.
>>
>> This is a very longstanding operational problem in Cassandra. Repair barely works and is opaque, yet one is expected to run it once a week in the default configuration.
>>
>> An unreasonably-hostile-in-tone-but-otherwise-accurate description of the status quo before the re-write of streaming in 2.0 :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-5396
>>
>> A proposal to change the default for gc_grace_seconds to 34 days, so that this fragile and heavyweight operation only has to be done once a month :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-5850
>>
>> granted , this is a lot of data, but it would be nice to at least see some progress.
>>
>> Here's the rewrite of streaming, where progress indication improves dramatically over the prior status quo :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-5286
>>
>> And here's two open tickets on making repair less opaque (thx yukim@#cassandra) :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-5483
>> https://issues.apache.org/jira/browse/CASSANDRA-5839
>>
>> =Rob
>>
>>
>
>
Re: stalled nodetool repair?
Posted by DuyHai Doan <do...@gmail.com>.
Thanks Ben for the link. Still this script does not work with vnodes, which
exclude a wide range of C* config
On Thu, Aug 21, 2014 at 5:51 PM, Ben Bromhead <be...@instaclustr.com> wrote:
> https://github.com/mstump/cassandra_range_repair
>
> Also very useful.
>
> Ben Bromhead
> Instaclustr | www.instaclustr.com | @instaclustr
> <http://twitter.com/instaclustr> | +61 415 936 359
>
>
>
>
> On 22/08/2014, at 6:12 AM, Robert Coli <rc...@eventbrite.com> wrote:
>
> On Thu, Aug 21, 2014 at 12:32 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> How do I watch the progress of nodetool repair.
>>
>
> This is a very longstanding operational problem in Cassandra. Repair
> barely works and is opaque, yet one is expected to run it once a week in
> the default configuration.
>
> An unreasonably-hostile-in-tone-but-otherwise-accurate description of the
> status quo before the re-write of streaming in 2.0 :
>
> https://issues.apache.org/jira/browse/CASSANDRA-5396
>
> A proposal to change the default for gc_grace_seconds to 34 days, so that
> this fragile and heavyweight operation only has to be done once a month :
>
> https://issues.apache.org/jira/browse/CASSANDRA-5850
>
>
>> granted , this is a lot of data, but it would be nice to at least see
>> some progress.
>>
>
> Here's the rewrite of streaming, where progress indication improves
> dramatically over the prior status quo :
>
> https://issues.apache.org/jira/browse/CASSANDRA-5286
>
> And here's two open tickets on making repair less opaque (thx yukim@#cassandra)
> :
>
> https://issues.apache.org/jira/browse/CASSANDRA-5483
> https://issues.apache.org/jira/browse/CASSANDRA-5839
>
> =Rob
>
>
>
>
Re: stalled nodetool repair?
Posted by Ben Bromhead <be...@instaclustr.com>.
https://github.com/mstump/cassandra_range_repair
Also very useful.
Ben Bromhead
Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359
On 22/08/2014, at 6:12 AM, Robert Coli <rc...@eventbrite.com> wrote:
> On Thu, Aug 21, 2014 at 12:32 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> How do I watch the progress of nodetool repair.
>
> This is a very longstanding operational problem in Cassandra. Repair barely works and is opaque, yet one is expected to run it once a week in the default configuration.
>
> An unreasonably-hostile-in-tone-but-otherwise-accurate description of the status quo before the re-write of streaming in 2.0 :
>
> https://issues.apache.org/jira/browse/CASSANDRA-5396
>
> A proposal to change the default for gc_grace_seconds to 34 days, so that this fragile and heavyweight operation only has to be done once a month :
>
> https://issues.apache.org/jira/browse/CASSANDRA-5850
>
> granted , this is a lot of data, but it would be nice to at least see some progress.
>
> Here's the rewrite of streaming, where progress indication improves dramatically over the prior status quo :
>
> https://issues.apache.org/jira/browse/CASSANDRA-5286
>
> And here's two open tickets on making repair less opaque (thx yukim@#cassandra) :
>
> https://issues.apache.org/jira/browse/CASSANDRA-5483
> https://issues.apache.org/jira/browse/CASSANDRA-5839
>
> =Rob
>
>
Re: stalled nodetool repair?
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Aug 21, 2014 at 12:32 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> How do I watch the progress of nodetool repair.
>
This is a very longstanding operational problem in Cassandra. Repair barely
works and is opaque, yet one is expected to run it once a week in the
default configuration.
An unreasonably-hostile-in-tone-but-otherwise-accurate description of the
status quo before the re-write of streaming in 2.0 :
https://issues.apache.org/jira/browse/CASSANDRA-5396
A proposal to change the default for gc_grace_seconds to 34 days, so that
this fragile and heavyweight operation only has to be done once a month :
https://issues.apache.org/jira/browse/CASSANDRA-5850
> granted , this is a lot of data, but it would be nice to at least see some
> progress.
>
Here's the rewrite of streaming, where progress indication improves
dramatically over the prior status quo :
https://issues.apache.org/jira/browse/CASSANDRA-5286
And here's two open tickets on making repair less opaque (thx yukim@#cassandra)
:
https://issues.apache.org/jira/browse/CASSANDRA-5483
https://issues.apache.org/jira/browse/CASSANDRA-5839
=Rob