You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Phil Burress <ph...@gmail.com> on 2014/06/30 23:54:08 UTC
nodetool repair -snapshot option?
We are running into an issue with nodetool repair. One or more of our nodes
will die with OOM errors when running nodetool repair on a single node. Was
reading this http://www.datastax.com/dev/blog/advanced-repair-techniques
and it mentioned using the -snapshot option, however, that doesn't appear
to be an option in the version we have. We are running 2.0.7 with vnodes.
Any insight into what might be causing these OOMs and/or what version this
-snapshot option is available in?
Thanks!
Phil
Re: nodetool repair -snapshot option?
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Jul 1, 2014 at 3:53 PM, Phil Burress <ph...@gmail.com>
wrote:
> Thanks! We retrieved all the ranges and started running repair on them. We
> ran through all of them but found one single range which brought the ENTIRE
> cluster down. All of the other ranges ran quickly and smoothly. This one
> problematic range reliably brings it down every time we try to run repair
> on it. Any thoughts on why one specific range would be a troublemaker?
>
Pathologically gigantic row?
=Rob
Re: nodetool repair -snapshot option?
Posted by Phil Burress <ph...@gmail.com>.
Thanks! We retrieved all the ranges and started running repair on them. We
ran through all of them but found one single range which brought the ENTIRE
cluster down. All of the other ranges ran quickly and smoothly. This one
problematic range reliably brings it down every time we try to run repair
on it. Any thoughts on why one specific range would be a troublemaker?
On Tue, Jul 1, 2014 at 11:44 AM, Ken Hancock <ke...@schange.com>
wrote:
> I also expanded on a script originally written by Matt Stump @ Datastax.
> The readme has the reasoning behind requiring sub-range repairs.
>
> https://github.com/hancockks/cassandra_range_repair
>
>
>
>
> On Mon, Jun 30, 2014 at 10:20 PM, Phil Burress <ph...@gmail.com>
> wrote:
>
>> @Paulo, this is very cool! Thanks very much for the link!
>>
>>
>> On Mon, Jun 30, 2014 at 9:37 PM, Paulo Ricardo Motta Gomes <
>> paulo.motta@chaordicsystems.com> wrote:
>>
>>> If you find it useful, I created a tool where you input the node IP,
>>> keyspace, column family, and optionally the number of partitions (default:
>>> 32K), and it outputs the list of subranges for that node, CF, partition
>>> size: https://github.com/pauloricardomg/cassandra-list-subranges
>>>
>>> So you can basically iterate over the output of that and do subrange
>>> repair for each node and cf, maybe in parallel. :)
>>>
>>>
>>> On Mon, Jun 30, 2014 at 10:26 PM, Phil Burress <philburresseme@gmail.com
>>> > wrote:
>>>
>>>> One last question. Any tips on scripting a subrange repair?
>>>>
>>>>
>>>> On Mon, Jun 30, 2014 at 7:12 PM, Phil Burress <philburresseme@gmail.com
>>>> > wrote:
>>>>
>>>>> We are running repair -pr. We've tried subrange manually and that
>>>>> seems to work ok. I guess we'll go with that going forward. Thanks for all
>>>>> the info!
>>>>>
>>>>>
>>>>> On Mon, Jun 30, 2014 at 6:52 PM, Jaydeep Chovatia <
>>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>>
>>>>>> Are you running full repair or on subset? If you are running full
>>>>>> repair then try running on sub-set of ranges which means less data to worry
>>>>>> during repair and that would help JAVA heap in general. You will have to do
>>>>>> multiple iterations to complete entire range but at-least it will work.
>>>>>>
>>>>>> -jaydeep
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 30, 2014 at 3:22 PM, Robert Coli <rc...@eventbrite.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Mon, Jun 30, 2014 at 3:08 PM, Yuki Morishita <mo...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Repair uses snapshot option by default since 2.0.2 (see NEWS.txt).
>>>>>>>>
>>>>>>>
>>>>>>> As a general meta comment, the process by which operationally
>>>>>>> important defaults change in Cassandra seems ad-hoc and sub-optimal.
>>>>>>>
>>>>>>> For to record, my view was that this change, which makes repair even
>>>>>>> slower than it previously was, was probably overly optimistic.
>>>>>>>
>>>>>>> It's also weird in that it changes default behavior which has been
>>>>>>> unchanged since the start of Cassandra time and is therefore probably
>>>>>>> automated against. Why was it so critically important to switch to snapshot
>>>>>>> repair that it needed to be shotgunned as a new default in 2.0.2?
>>>>>>>
>>>>>>> =Rob
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Paulo Motta*
>>>
>>> Chaordic | *Platform*
>>> *www.chaordic.com.br <http://www.chaordic.com.br/>*
>>> +55 48 3232.3200
>>>
>>
>>
>
>
> --
> *Ken Hancock *| System Architect, Advanced Advertising
> SeaChange International
> 50 Nagog Park
> Acton, Massachusetts 01720
> ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC
> <http://www.schange.com/en-US/Company/InvestorRelations.aspx>
> Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hancock@schange.com
> | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image:
> LinkedIn] <http://www.linkedin.com/in/kenhancock>
>
> [image: SeaChange International]
> <http://www.schange.com/>This e-mail and any attachments may contain
> information which is SeaChange International confidential. The information
> enclosed is intended only for the addressees herein and may not be copied
> or forwarded without permission from SeaChange International.
>
Re: nodetool repair -snapshot option?
Posted by Ken Hancock <ke...@schange.com>.
I also expanded on a script originally written by Matt Stump @ Datastax.
The readme has the reasoning behind requiring sub-range repairs.
https://github.com/hancockks/cassandra_range_repair
On Mon, Jun 30, 2014 at 10:20 PM, Phil Burress <ph...@gmail.com>
wrote:
> @Paulo, this is very cool! Thanks very much for the link!
>
>
> On Mon, Jun 30, 2014 at 9:37 PM, Paulo Ricardo Motta Gomes <
> paulo.motta@chaordicsystems.com> wrote:
>
>> If you find it useful, I created a tool where you input the node IP,
>> keyspace, column family, and optionally the number of partitions (default:
>> 32K), and it outputs the list of subranges for that node, CF, partition
>> size: https://github.com/pauloricardomg/cassandra-list-subranges
>>
>> So you can basically iterate over the output of that and do subrange
>> repair for each node and cf, maybe in parallel. :)
>>
>>
>> On Mon, Jun 30, 2014 at 10:26 PM, Phil Burress <ph...@gmail.com>
>> wrote:
>>
>>> One last question. Any tips on scripting a subrange repair?
>>>
>>>
>>> On Mon, Jun 30, 2014 at 7:12 PM, Phil Burress <ph...@gmail.com>
>>> wrote:
>>>
>>>> We are running repair -pr. We've tried subrange manually and that seems
>>>> to work ok. I guess we'll go with that going forward. Thanks for all the
>>>> info!
>>>>
>>>>
>>>> On Mon, Jun 30, 2014 at 6:52 PM, Jaydeep Chovatia <
>>>> chovatia.jaydeep@gmail.com> wrote:
>>>>
>>>>> Are you running full repair or on subset? If you are running full
>>>>> repair then try running on sub-set of ranges which means less data to worry
>>>>> during repair and that would help JAVA heap in general. You will have to do
>>>>> multiple iterations to complete entire range but at-least it will work.
>>>>>
>>>>> -jaydeep
>>>>>
>>>>>
>>>>> On Mon, Jun 30, 2014 at 3:22 PM, Robert Coli <rc...@eventbrite.com>
>>>>> wrote:
>>>>>
>>>>>> On Mon, Jun 30, 2014 at 3:08 PM, Yuki Morishita <mo...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Repair uses snapshot option by default since 2.0.2 (see NEWS.txt).
>>>>>>>
>>>>>>
>>>>>> As a general meta comment, the process by which operationally
>>>>>> important defaults change in Cassandra seems ad-hoc and sub-optimal.
>>>>>>
>>>>>> For to record, my view was that this change, which makes repair even
>>>>>> slower than it previously was, was probably overly optimistic.
>>>>>>
>>>>>> It's also weird in that it changes default behavior which has been
>>>>>> unchanged since the start of Cassandra time and is therefore probably
>>>>>> automated against. Why was it so critically important to switch to snapshot
>>>>>> repair that it needed to be shotgunned as a new default in 2.0.2?
>>>>>>
>>>>>> =Rob
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> *Paulo Motta*
>>
>> Chaordic | *Platform*
>> *www.chaordic.com.br <http://www.chaordic.com.br/>*
>> +55 48 3232.3200
>>
>
>
--
*Ken Hancock *| System Architect, Advanced Advertising
SeaChange International
50 Nagog Park
Acton, Massachusetts 01720
ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC
<http://www.schange.com/en-US/Company/InvestorRelations.aspx>
Office: +1 (978) 889-3329 | [image: Google Talk:]
ken.hancock@schange.com | [image:
Skype:]hancockks | [image: Yahoo IM:]hancockks [image: LinkedIn]
<http://www.linkedin.com/in/kenhancock>
[image: SeaChange International]
<http://www.schange.com/>This e-mail and any attachments may contain
information which is SeaChange International confidential. The information
enclosed is intended only for the addressees herein and may not be copied
or forwarded without permission from SeaChange International.
Re: nodetool repair -snapshot option?
Posted by Phil Burress <ph...@gmail.com>.
@Paulo, this is very cool! Thanks very much for the link!
On Mon, Jun 30, 2014 at 9:37 PM, Paulo Ricardo Motta Gomes <
paulo.motta@chaordicsystems.com> wrote:
> If you find it useful, I created a tool where you input the node IP,
> keyspace, column family, and optionally the number of partitions (default:
> 32K), and it outputs the list of subranges for that node, CF, partition
> size: https://github.com/pauloricardomg/cassandra-list-subranges
>
> So you can basically iterate over the output of that and do subrange
> repair for each node and cf, maybe in parallel. :)
>
>
> On Mon, Jun 30, 2014 at 10:26 PM, Phil Burress <ph...@gmail.com>
> wrote:
>
>> One last question. Any tips on scripting a subrange repair?
>>
>>
>> On Mon, Jun 30, 2014 at 7:12 PM, Phil Burress <ph...@gmail.com>
>> wrote:
>>
>>> We are running repair -pr. We've tried subrange manually and that seems
>>> to work ok. I guess we'll go with that going forward. Thanks for all the
>>> info!
>>>
>>>
>>> On Mon, Jun 30, 2014 at 6:52 PM, Jaydeep Chovatia <
>>> chovatia.jaydeep@gmail.com> wrote:
>>>
>>>> Are you running full repair or on subset? If you are running full
>>>> repair then try running on sub-set of ranges which means less data to worry
>>>> during repair and that would help JAVA heap in general. You will have to do
>>>> multiple iterations to complete entire range but at-least it will work.
>>>>
>>>> -jaydeep
>>>>
>>>>
>>>> On Mon, Jun 30, 2014 at 3:22 PM, Robert Coli <rc...@eventbrite.com>
>>>> wrote:
>>>>
>>>>> On Mon, Jun 30, 2014 at 3:08 PM, Yuki Morishita <mo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Repair uses snapshot option by default since 2.0.2 (see NEWS.txt).
>>>>>>
>>>>>
>>>>> As a general meta comment, the process by which operationally
>>>>> important defaults change in Cassandra seems ad-hoc and sub-optimal.
>>>>>
>>>>> For to record, my view was that this change, which makes repair even
>>>>> slower than it previously was, was probably overly optimistic.
>>>>>
>>>>> It's also weird in that it changes default behavior which has been
>>>>> unchanged since the start of Cassandra time and is therefore probably
>>>>> automated against. Why was it so critically important to switch to snapshot
>>>>> repair that it needed to be shotgunned as a new default in 2.0.2?
>>>>>
>>>>> =Rob
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> *Paulo Motta*
>
> Chaordic | *Platform*
> *www.chaordic.com.br <http://www.chaordic.com.br/>*
> +55 48 3232.3200
>
Re: nodetool repair -snapshot option?
Posted by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com>.
If you find it useful, I created a tool where you input the node IP,
keyspace, column family, and optionally the number of partitions (default:
32K), and it outputs the list of subranges for that node, CF, partition
size: https://github.com/pauloricardomg/cassandra-list-subranges
So you can basically iterate over the output of that and do subrange repair
for each node and cf, maybe in parallel. :)
On Mon, Jun 30, 2014 at 10:26 PM, Phil Burress <ph...@gmail.com>
wrote:
> One last question. Any tips on scripting a subrange repair?
>
>
> On Mon, Jun 30, 2014 at 7:12 PM, Phil Burress <ph...@gmail.com>
> wrote:
>
>> We are running repair -pr. We've tried subrange manually and that seems
>> to work ok. I guess we'll go with that going forward. Thanks for all the
>> info!
>>
>>
>> On Mon, Jun 30, 2014 at 6:52 PM, Jaydeep Chovatia <
>> chovatia.jaydeep@gmail.com> wrote:
>>
>>> Are you running full repair or on subset? If you are running full repair
>>> then try running on sub-set of ranges which means less data to worry during
>>> repair and that would help JAVA heap in general. You will have to do
>>> multiple iterations to complete entire range but at-least it will work.
>>>
>>> -jaydeep
>>>
>>>
>>> On Mon, Jun 30, 2014 at 3:22 PM, Robert Coli <rc...@eventbrite.com>
>>> wrote:
>>>
>>>> On Mon, Jun 30, 2014 at 3:08 PM, Yuki Morishita <mo...@gmail.com>
>>>> wrote:
>>>>
>>>>> Repair uses snapshot option by default since 2.0.2 (see NEWS.txt).
>>>>>
>>>>
>>>> As a general meta comment, the process by which operationally important
>>>> defaults change in Cassandra seems ad-hoc and sub-optimal.
>>>>
>>>> For to record, my view was that this change, which makes repair even
>>>> slower than it previously was, was probably overly optimistic.
>>>>
>>>> It's also weird in that it changes default behavior which has been
>>>> unchanged since the start of Cassandra time and is therefore probably
>>>> automated against. Why was it so critically important to switch to snapshot
>>>> repair that it needed to be shotgunned as a new default in 2.0.2?
>>>>
>>>> =Rob
>>>>
>>>>
>>>
>>>
>>
>
--
*Paulo Motta*
Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200
Re: nodetool repair -snapshot option?
Posted by Phil Burress <ph...@gmail.com>.
One last question. Any tips on scripting a subrange repair?
On Mon, Jun 30, 2014 at 7:12 PM, Phil Burress <ph...@gmail.com>
wrote:
> We are running repair -pr. We've tried subrange manually and that seems to
> work ok. I guess we'll go with that going forward. Thanks for all the info!
>
>
> On Mon, Jun 30, 2014 at 6:52 PM, Jaydeep Chovatia <
> chovatia.jaydeep@gmail.com> wrote:
>
>> Are you running full repair or on subset? If you are running full repair
>> then try running on sub-set of ranges which means less data to worry during
>> repair and that would help JAVA heap in general. You will have to do
>> multiple iterations to complete entire range but at-least it will work.
>>
>> -jaydeep
>>
>>
>> On Mon, Jun 30, 2014 at 3:22 PM, Robert Coli <rc...@eventbrite.com>
>> wrote:
>>
>>> On Mon, Jun 30, 2014 at 3:08 PM, Yuki Morishita <mo...@gmail.com>
>>> wrote:
>>>
>>>> Repair uses snapshot option by default since 2.0.2 (see NEWS.txt).
>>>>
>>>
>>> As a general meta comment, the process by which operationally important
>>> defaults change in Cassandra seems ad-hoc and sub-optimal.
>>>
>>> For to record, my view was that this change, which makes repair even
>>> slower than it previously was, was probably overly optimistic.
>>>
>>> It's also weird in that it changes default behavior which has been
>>> unchanged since the start of Cassandra time and is therefore probably
>>> automated against. Why was it so critically important to switch to snapshot
>>> repair that it needed to be shotgunned as a new default in 2.0.2?
>>>
>>> =Rob
>>>
>>>
>>
>>
>
Re: nodetool repair -snapshot option?
Posted by Phil Burress <ph...@gmail.com>.
We are running repair -pr. We've tried subrange manually and that seems to
work ok. I guess we'll go with that going forward. Thanks for all the info!
On Mon, Jun 30, 2014 at 6:52 PM, Jaydeep Chovatia <
chovatia.jaydeep@gmail.com> wrote:
> Are you running full repair or on subset? If you are running full repair
> then try running on sub-set of ranges which means less data to worry during
> repair and that would help JAVA heap in general. You will have to do
> multiple iterations to complete entire range but at-least it will work.
>
> -jaydeep
>
>
> On Mon, Jun 30, 2014 at 3:22 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Mon, Jun 30, 2014 at 3:08 PM, Yuki Morishita <mo...@gmail.com>
>> wrote:
>>
>>> Repair uses snapshot option by default since 2.0.2 (see NEWS.txt).
>>>
>>
>> As a general meta comment, the process by which operationally important
>> defaults change in Cassandra seems ad-hoc and sub-optimal.
>>
>> For to record, my view was that this change, which makes repair even
>> slower than it previously was, was probably overly optimistic.
>>
>> It's also weird in that it changes default behavior which has been
>> unchanged since the start of Cassandra time and is therefore probably
>> automated against. Why was it so critically important to switch to snapshot
>> repair that it needed to be shotgunned as a new default in 2.0.2?
>>
>> =Rob
>>
>>
>
>
Re: nodetool repair -snapshot option?
Posted by Jaydeep Chovatia <ch...@gmail.com>.
Are you running full repair or on subset? If you are running full repair
then try running on sub-set of ranges which means less data to worry during
repair and that would help JAVA heap in general. You will have to do
multiple iterations to complete entire range but at-least it will work.
-jaydeep
On Mon, Jun 30, 2014 at 3:22 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Jun 30, 2014 at 3:08 PM, Yuki Morishita <mo...@gmail.com>
> wrote:
>
>> Repair uses snapshot option by default since 2.0.2 (see NEWS.txt).
>>
>
> As a general meta comment, the process by which operationally important
> defaults change in Cassandra seems ad-hoc and sub-optimal.
>
> For to record, my view was that this change, which makes repair even
> slower than it previously was, was probably overly optimistic.
>
> It's also weird in that it changes default behavior which has been
> unchanged since the start of Cassandra time and is therefore probably
> automated against. Why was it so critically important to switch to snapshot
> repair that it needed to be shotgunned as a new default in 2.0.2?
>
> =Rob
>
>
Re: nodetool repair -snapshot option?
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Jun 30, 2014 at 3:08 PM, Yuki Morishita <mo...@gmail.com> wrote:
> Repair uses snapshot option by default since 2.0.2 (see NEWS.txt).
>
As a general meta comment, the process by which operationally important
defaults change in Cassandra seems ad-hoc and sub-optimal.
For to record, my view was that this change, which makes repair even slower
than it previously was, was probably overly optimistic.
It's also weird in that it changes default behavior which has been
unchanged since the start of Cassandra time and is therefore probably
automated against. Why was it so critically important to switch to snapshot
repair that it needed to be shotgunned as a new default in 2.0.2?
=Rob
Re: nodetool repair -snapshot option?
Posted by Kevin Burton <bu...@spinn3r.com>.
The stack won't help a ton since the memory leak will occur elsewhere… the
stack will just have the point where the memory allocation failed :-(
On Mon, Jun 30, 2014 at 3:08 PM, Yuki Morishita <mo...@gmail.com> wrote:
> Repair uses snapshot option by default since 2.0.2 (see NEWS.txt).
> So you don't have to specify in your version.
>
> Do you have stacktrace when OOMed?
>
> On Mon, Jun 30, 2014 at 4:54 PM, Phil Burress <ph...@gmail.com>
> wrote:
> > We are running into an issue with nodetool repair. One or more of our
> nodes
> > will die with OOM errors when running nodetool repair on a single node.
> Was
> > reading this http://www.datastax.com/dev/blog/advanced-repair-techniques
> and
> > it mentioned using the -snapshot option, however, that doesn't appear to
> be
> > an option in the version we have. We are running 2.0.7 with vnodes. Any
> > insight into what might be causing these OOMs and/or what version this
> > -snapshot option is available in?
> >
> > Thanks!
> >
> > Phil
>
>
>
> --
> Yuki Morishita
> t:yukim (http://twitter.com/yukim)
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: nodetool repair -snapshot option?
Posted by Yuki Morishita <mo...@gmail.com>.
Repair uses snapshot option by default since 2.0.2 (see NEWS.txt).
So you don't have to specify in your version.
Do you have stacktrace when OOMed?
On Mon, Jun 30, 2014 at 4:54 PM, Phil Burress <ph...@gmail.com> wrote:
> We are running into an issue with nodetool repair. One or more of our nodes
> will die with OOM errors when running nodetool repair on a single node. Was
> reading this http://www.datastax.com/dev/blog/advanced-repair-techniques and
> it mentioned using the -snapshot option, however, that doesn't appear to be
> an option in the version we have. We are running 2.0.7 with vnodes. Any
> insight into what might be causing these OOMs and/or what version this
> -snapshot option is available in?
>
> Thanks!
>
> Phil
--
Yuki Morishita
t:yukim (http://twitter.com/yukim)