You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Mohapatra, Kishore" <Ki...@nuance.com> on 2017/10/04 16:46:28 UTC
Increasing VNodes
Hi,
We are having a lot of problems in repair process. We use sub range repair. But most of the time, some ranges fails with streaming error or some other kind of error.
So wondering if it will help if we increase the no. of VNodes from 256 (default) to 512. But increasing the VNodes will be a lot of efforts, as it involves wiping out the data and bootstrapping.
So is there any other way of splitting the range into small ranges ?
We are using version 2.1.15.4 at the moment.
Thanks
Kishore Mohapatra
Principal Operations DBA
Seattle, WA
Email : kishore.mohapatra@nuance.com<ma...@nuance.com>
Re: [EXTERNAL] Re: Increasing VNodes
Posted by Chris Lohfink <cl...@gmail.com>.
Cant you just increase segmentCount option to split it more?
On Wed, Oct 4, 2017 at 12:50 PM, Mohapatra, Kishore <
Kishore.Mohapatra@nuance.com> wrote:
> Thanks a lot for all of your input. We are actually using Cassandra
> reaper. But it is just splitting the ranges into 256 per node.
>
> But I will certainly try out splitting into smaller ranges going thru the
> system.size_estimate table.
>
>
>
> Thanks
>
>
>
> *Kishore Mohapatra*
>
> Principal Operations DBA
>
> Seattle, WA
>
> Email : kishore.mohapatra@nuance.com
>
>
>
>
>
> *From:* Jon Haddad [mailto:jonathan.haddad@gmail.com] * On Behalf Of *Jon
> Haddad
> *Sent:* Wednesday, October 04, 2017 10:27 AM
> *To:* user <us...@cassandra.apache.org>
> *Subject:* [EXTERNAL] Re: Increasing VNodes
>
>
>
> The site (with the docs) is probably more helpful to learn about how
> reaper works: http://cassandra-reaper.io/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra-2Dreaper.io_&d=DwMFAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=O20_rcIS1QazTO3_J10I1cPIygxnuBZ4sUCz1TS16XE&m=nHN7toaSQUjfwSABx1KXlVHLYmlaEcUMYPHzC3ky5TM&s=oqOGMK4a6er4kSRtzs7B2_A6QB6kb7nQek8NAU5pytI&e=>
>
>
>
> On Oct 4, 2017, at 9:54 AM, Chris Lohfink <cl...@gmail.com> wrote:
>
>
>
> Increasing number of tokens will make repairs worse not better. You can
> just split the sub ranges into smaller chunks, you dont need to use vnodes
> to do that. Simple approach is to iterate through each host token range and
> split by N and repair them (ie https://github.com/onzra/
> cassandra_range_repair
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_onzra_cassandra-5Frange-5Frepair&d=DwMFAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=O20_rcIS1QazTO3_J10I1cPIygxnuBZ4sUCz1TS16XE&m=nHN7toaSQUjfwSABx1KXlVHLYmlaEcUMYPHzC3ky5TM&s=Ph50r9wV17T72OEwI3FsbAXBVZ3Pt-AmACQZYdsQqgk&e=>)
> To be more efficient you can grab ranges and split based on number of
> partitions in the range (ie fetch system.size_estimates and walk that) so
> you dont split empty or small ranges a ton unnecessarily, and because not
> all tables have some fixed N that is efficient.
>
>
>
> Using TLPs reaper https://github.com/thelastpickle/cassandra-reaper
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_thelastpickle_cassandra-2Dreaper&d=DwMFAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=O20_rcIS1QazTO3_J10I1cPIygxnuBZ4sUCz1TS16XE&m=nHN7toaSQUjfwSABx1KXlVHLYmlaEcUMYPHzC3ky5TM&s=_4VgTSxIgqGn339jpHMycnHg4bmM0pHmUxSQ8nNfdDU&e=> or
> DataStax OpsCenter's repair service is easiest solution without a lot of
> effort. Repairs are hard.
>
>
>
> Chris
>
>
>
> On Wed, Oct 4, 2017 at 11:48 AM, Jeff Jirsa <jj...@gmail.com> wrote:
>
> You don't need to change the number of vnodes, you can manually select
> CONTAINED token subranges and pass in -st and -et (just try to pick a
> number > 2^20 that is fully contained by at least one vnode).
>
>
>
>
>
>
>
>
>
> On Wed, Oct 4, 2017 at 9:46 AM, Mohapatra, Kishore <
> Kishore.Mohapatra@nuance.com> wrote:
>
> Hi,
>
> We are having a lot of problems in repair process. We use sub
> range repair. But most of the time, some ranges fails with streaming error
> or some other kind of error.
>
> So wondering if it will help if we increase the no. of VNodes from 256
> (default) to 512. But increasing the VNodes will be a lot of efforts, as it
> involves wiping out the data and bootstrapping.
>
> So is there any other way of splitting the range into small ranges ?
>
>
>
> We are using version 2.1.15.4 at the moment.
>
>
>
> Thanks
>
>
>
> *Kishore Mohapatra*
>
> Principal Operations DBA
>
> Seattle, WA
>
> Email : kishore.mohapatra@nuance.com
>
>
>
>
>
>
>
>
>
>
>
RE: [EXTERNAL] Re: Increasing VNodes
Posted by "Mohapatra, Kishore" <Ki...@nuance.com>.
Thanks a lot for all of your input. We are actually using Cassandra reaper. But it is just splitting the ranges into 256 per node.
But I will certainly try out splitting into smaller ranges going thru the system.size_estimate table.
Thanks
Kishore Mohapatra
Principal Operations DBA
Seattle, WA
Email : kishore.mohapatra@nuance.com<ma...@nuance.com>
From: Jon Haddad [mailto:jonathan.haddad@gmail.com] On Behalf Of Jon Haddad
Sent: Wednesday, October 04, 2017 10:27 AM
To: user <us...@cassandra.apache.org>
Subject: [EXTERNAL] Re: Increasing VNodes
The site (with the docs) is probably more helpful to learn about how reaper works: http://cassandra-reaper.io/<https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra-2Dreaper.io_&d=DwMFAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=O20_rcIS1QazTO3_J10I1cPIygxnuBZ4sUCz1TS16XE&m=nHN7toaSQUjfwSABx1KXlVHLYmlaEcUMYPHzC3ky5TM&s=oqOGMK4a6er4kSRtzs7B2_A6QB6kb7nQek8NAU5pytI&e=>
On Oct 4, 2017, at 9:54 AM, Chris Lohfink <cl...@gmail.com>> wrote:
Increasing number of tokens will make repairs worse not better. You can just split the sub ranges into smaller chunks, you dont need to use vnodes to do that. Simple approach is to iterate through each host token range and split by N and repair them (ie https://github.com/onzra/cassandra_range_repair<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_onzra_cassandra-5Frange-5Frepair&d=DwMFAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=O20_rcIS1QazTO3_J10I1cPIygxnuBZ4sUCz1TS16XE&m=nHN7toaSQUjfwSABx1KXlVHLYmlaEcUMYPHzC3ky5TM&s=Ph50r9wV17T72OEwI3FsbAXBVZ3Pt-AmACQZYdsQqgk&e=>) To be more efficient you can grab ranges and split based on number of partitions in the range (ie fetch system.size_estimates and walk that) so you dont split empty or small ranges a ton unnecessarily, and because not all tables have some fixed N that is efficient.
Using TLPs reaper https://github.com/thelastpickle/cassandra-reaper<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_thelastpickle_cassandra-2Dreaper&d=DwMFAg&c=djjh8EKwHtOepW4Bjau0lKhLlu-DxM1dlgP0rrLsOzY&r=O20_rcIS1QazTO3_J10I1cPIygxnuBZ4sUCz1TS16XE&m=nHN7toaSQUjfwSABx1KXlVHLYmlaEcUMYPHzC3ky5TM&s=_4VgTSxIgqGn339jpHMycnHg4bmM0pHmUxSQ8nNfdDU&e=> or DataStax OpsCenter's repair service is easiest solution without a lot of effort. Repairs are hard.
Chris
On Wed, Oct 4, 2017 at 11:48 AM, Jeff Jirsa <jj...@gmail.com>> wrote:
You don't need to change the number of vnodes, you can manually select CONTAINED token subranges and pass in -st and -et (just try to pick a number > 2^20 that is fully contained by at least one vnode).
On Wed, Oct 4, 2017 at 9:46 AM, Mohapatra, Kishore <Ki...@nuance.com>> wrote:
Hi,
We are having a lot of problems in repair process. We use sub range repair. But most of the time, some ranges fails with streaming error or some other kind of error.
So wondering if it will help if we increase the no. of VNodes from 256 (default) to 512. But increasing the VNodes will be a lot of efforts, as it involves wiping out the data and bootstrapping.
So is there any other way of splitting the range into small ranges ?
We are using version 2.1.15.4 at the moment.
Thanks
Kishore Mohapatra
Principal Operations DBA
Seattle, WA
Email : kishore.mohapatra@nuance.com<ma...@nuance.com>
Re: Increasing VNodes
Posted by Jon Haddad <jo...@jonhaddad.com>.
The site (with the docs) is probably more helpful to learn about how reaper works: http://cassandra-reaper.io/ <http://cassandra-reaper.io/>
> On Oct 4, 2017, at 9:54 AM, Chris Lohfink <cl...@gmail.com> wrote:
>
> Increasing number of tokens will make repairs worse not better. You can just split the sub ranges into smaller chunks, you dont need to use vnodes to do that. Simple approach is to iterate through each host token range and split by N and repair them (ie https://github.com/onzra/cassandra_range_repair <https://github.com/onzra/cassandra_range_repair>) To be more efficient you can grab ranges and split based on number of partitions in the range (ie fetch system.size_estimates and walk that) so you dont split empty or small ranges a ton unnecessarily, and because not all tables have some fixed N that is efficient.
>
> Using TLPs reaper https://github.com/thelastpickle/cassandra-reaper <https://github.com/thelastpickle/cassandra-reaper> or DataStax OpsCenter's repair service is easiest solution without a lot of effort. Repairs are hard.
>
> Chris
>
> On Wed, Oct 4, 2017 at 11:48 AM, Jeff Jirsa <jjirsa@gmail.com <ma...@gmail.com>> wrote:
> You don't need to change the number of vnodes, you can manually select CONTAINED token subranges and pass in -st and -et (just try to pick a number > 2^20 that is fully contained by at least one vnode).
>
>
>
>
> On Wed, Oct 4, 2017 at 9:46 AM, Mohapatra, Kishore <Kishore.Mohapatra@nuance.com <ma...@nuance.com>> wrote:
> Hi,
>
> We are having a lot of problems in repair process. We use sub range repair. But most of the time, some ranges fails with streaming error or some other kind of error.
>
> So wondering if it will help if we increase the no. of VNodes from 256 (default) to 512. But increasing the VNodes will be a lot of efforts, as it involves wiping out the data and bootstrapping.
>
> So is there any other way of splitting the range into small ranges ?
>
>
>
> We are using version 2.1.15.4 at the moment.
>
>
>
> Thanks
>
>
>
> Kishore Mohapatra
>
> Principal Operations DBA
>
> Seattle, WA
>
> Email : kishore.mohapatra@nuance.com <ma...@nuance.com>
>
>
>
>
>
>
Re: Increasing VNodes
Posted by Chris Lohfink <cl...@gmail.com>.
Increasing number of tokens will make repairs worse not better. You can
just split the sub ranges into smaller chunks, you dont need to use vnodes
to do that. Simple approach is to iterate through each host token range and
split by N and repair them (ie
https://github.com/onzra/cassandra_range_repair) To be more efficient you
can grab ranges and split based on number of partitions in the range (ie
fetch system.size_estimates and walk that) so you dont split empty or small
ranges a ton unnecessarily, and because not all tables have some fixed N
that is efficient.
Using TLPs reaper https://github.com/thelastpickle/cassandra-reaper or
DataStax OpsCenter's repair service is easiest solution without a lot of
effort. Repairs are hard.
Chris
On Wed, Oct 4, 2017 at 11:48 AM, Jeff Jirsa <jj...@gmail.com> wrote:
> You don't need to change the number of vnodes, you can manually select
> CONTAINED token subranges and pass in -st and -et (just try to pick a
> number > 2^20 that is fully contained by at least one vnode).
>
>
>
>
> On Wed, Oct 4, 2017 at 9:46 AM, Mohapatra, Kishore <
> Kishore.Mohapatra@nuance.com> wrote:
>
>> Hi,
>>
>> We are having a lot of problems in repair process. We use sub
>> range repair. But most of the time, some ranges fails with streaming error
>> or some other kind of error.
>>
>> So wondering if it will help if we increase the no. of VNodes from 256
>> (default) to 512. But increasing the VNodes will be a lot of efforts, as it
>> involves wiping out the data and bootstrapping.
>>
>> So is there any other way of splitting the range into small ranges ?
>>
>>
>>
>> We are using version 2.1.15.4 at the moment.
>>
>>
>>
>> Thanks
>>
>>
>>
>> *Kishore Mohapatra*
>>
>> Principal Operations DBA
>>
>> Seattle, WA
>>
>> Email : kishore.mohapatra@nuance.com
>>
>>
>>
>>
>>
>
>
Re: Increasing VNodes
Posted by Jeff Jirsa <jj...@gmail.com>.
You don't need to change the number of vnodes, you can manually select
CONTAINED token subranges and pass in -st and -et (just try to pick a
number > 2^20 that is fully contained by at least one vnode).
On Wed, Oct 4, 2017 at 9:46 AM, Mohapatra, Kishore <
Kishore.Mohapatra@nuance.com> wrote:
> Hi,
>
> We are having a lot of problems in repair process. We use sub
> range repair. But most of the time, some ranges fails with streaming error
> or some other kind of error.
>
> So wondering if it will help if we increase the no. of VNodes from 256
> (default) to 512. But increasing the VNodes will be a lot of efforts, as it
> involves wiping out the data and bootstrapping.
>
> So is there any other way of splitting the range into small ranges ?
>
>
>
> We are using version 2.1.15.4 at the moment.
>
>
>
> Thanks
>
>
>
> *Kishore Mohapatra*
>
> Principal Operations DBA
>
> Seattle, WA
>
> Email : kishore.mohapatra@nuance.com
>
>
>
>
>