You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Kevin Burton <bu...@spinn3r.com> on 2015/10/19 02:18:21 UTC
compact/repair shouldn't compete for normal compaction resources.
I'm doing a big nodetool repair right now and I'm pretty sure the added
overhead is impacting our performance.
Shouldn't you be able to throttle repair so that normal compactions can use
most of the resources?
--
We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: compact/repair shouldn't compete for normal compaction resources.
Posted by Kevin Burton <bu...@spinn3r.com>.
Yes.. .it's not currently possible :)
I think it should be.
Say the IO on your C* is at 60% utilization.
If you do a repair, this would require 120% utilization obviously not
possible, so now your app is down / offline until the repair finishes.
If you could throttle repair separately this would resolve this problem.
IF anyone else thinks this is an issue I'll create a JIRA.
On Mon, Oct 19, 2015 at 3:38 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Oct 19, 2015 at 9:30 AM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> I think the point I was trying to make is that on highly loaded boxes,
>> repair should take lower priority than normal compactions.
>>
>
> You can manually do this by changing the thread priority of compaction
> threads which you somhow identify as doing repair related compaction...
>
> ... but incoming streamed SStables are compacted just as if they were
> flushed, so I'm pretty sure what you're asking for is not currently
> possible?
>
> =Rob
>
>
--
We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: compact/repair shouldn't compete for normal compaction resources.
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Oct 19, 2015 at 9:30 AM, Kevin Burton <bu...@spinn3r.com> wrote:
> I think the point I was trying to make is that on highly loaded boxes,
> repair should take lower priority than normal compactions.
>
You can manually do this by changing the thread priority of compaction
threads which you somhow identify as doing repair related compaction...
... but incoming streamed SStables are compacted just as if they were
flushed, so I'm pretty sure what you're asking for is not currently
possible?
=Rob
Re: compact/repair shouldn't compete for normal compaction resources.
Posted by Kevin Burton <bu...@spinn3r.com>.
I think the point I was trying to make is that on highly loaded boxes,
repair should take lower priority than normal compactions.
Having a throttle on *both* doesn't solve the problem.
So I need a
setcompactionthroughput
and a
setrepairthroughput
and total througput would be the sum of both.
On Mon, Oct 19, 2015 at 8:30 AM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:
> The validation compaction part of repair is susceptible to the compaction
> throttling knob `nodetool getcompactionthroughput`
> / `nodetool setcompactionthroughput` and you can use that to tune down the
> resources that are being used by repair.
>
> Check out this post by driftx on advanced repair techniques
> <http://www.datastax.com/dev/blog/advanced-repair-techniques>.
>
> Given your other question, I agree with Raj that it might be a good idea
> to decommission the new nodes rather than repairing depending on how much
> data has made it to them and how tight you were on resources before adding
> nodes.
>
>
> All the best,
>
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
> <http://goog_410786983>
>
>
> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Sun, Oct 18, 2015 at 8:18 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> I'm doing a big nodetool repair right now and I'm pretty sure the added
>> overhead is impacting our performance.
>>
>> Shouldn't you be able to throttle repair so that normal compactions can
>> use most of the resources?
>>
>> --
>>
>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>> Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>>
>>
>
--
We’re hiring if you know of any awesome Java Devops or Linux Operations
Engineers!
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
Re: compact/repair shouldn't compete for normal compaction resources.
Posted by Sebastian Estevez <se...@datastax.com>.
The validation compaction part of repair is susceptible to the compaction
throttling knob `nodetool getcompactionthroughput`
/ `nodetool setcompactionthroughput` and you can use that to tune down the
resources that are being used by repair.
Check out this post by driftx on advanced repair techniques
<http://www.datastax.com/dev/blog/advanced-repair-techniques>.
Given your other question, I agree with Raj that it might be a good idea to
decommission the new nodes rather than repairing depending on how much data
has made it to them and how tight you were on resources before adding nodes.
All the best,
[image: datastax_logo.png] <http://www.datastax.com/>
Sebastián Estévez
Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://goog_410786983>
<http://www.datastax.com/gartner-magic-quadrant-odbms>
DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.
On Sun, Oct 18, 2015 at 8:18 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> I'm doing a big nodetool repair right now and I'm pretty sure the added
> overhead is impacting our performance.
>
> Shouldn't you be able to throttle repair so that normal compactions can
> use most of the resources?
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
>
>