You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Terje Marthinussen <tm...@gmail.com> on 2011/05/07 11:01:52 UTC
compaction strategy
Even with the current concurrent compactions, given a high speed datafeed,
compactions will obviously start lagging at some stage, and once it does,
things can turn bad in terms of disk usage and read performance.
I have not read the compaction code well, but if
http://wiki.apache.org/cassandra/MemtableSSTable is up to date, I am
wondering if it:
1. Would it make sense to make full compactions occur a bit more aggressive.
That is, regardless of sstables of matching sizes, if the total number of
outstanding sstables gets above a certain datasize, would it make sense to
just schedule a full compaction rather than go through all the hoops of
gradually merging them in groups of matching sizes?
2. This is same topic, just another viewpoint. When you get to the stage
that compactionstats shows you something crazy like "pending tasks: 600", I
would think the code should be smart enough to either trigger a full
compaction and scrap the current queue, or at least merge some of those
pending tasks into larger ones instead of reading and writing the same data
again and again gradually merging it into larger and larger sizes?
The target behind both 1 and 2 would be to reduce the number of time data is
re-read and re-written in compactions before you reach the full dataset
size.
Re: compaction strategy
Posted by Peter Schuller <pe...@infidyne.com>.
> If you are seeing 600 pending compaction tasks regularly you almost
> definitely need more hardware.
Note that pending compactions is pretty misleading and you can't
really draw conclusions just based on the pending compactions
number/graph. For example, standard behavior during e.g.a long repair
may end up accumulating thousands of pending compactions that suddenly
drop to zero once you're done and a bunch of tasks that don't actually
need to do anything are "completed". With the concurrent compaction
support I suppose this will be mitigated as long as you don't hit your
concurrency limit.
--
/ Peter Schuller
Re: compaction strategy
Posted by Edward Capriolo <ed...@gmail.com>.
On Sat, May 7, 2011 at 8:54 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> On Sat, May 7, 2011 at 2:01 AM, Terje Marthinussen
> <tm...@gmail.com> wrote:
>> 1. Would it make sense to make full compactions occur a bit more aggressive.
>
> I'd rather reduce the performance impact of being behind, than do more
> full compactions: https://issues.apache.org/jira/browse/CASSANDRA-2498
>
>> 2. I
>> would think the code should be smart enough to either trigger a full
>> compaction and scrap the current queue, or at least merge some of those
>> pending tasks into larger ones
>
> Not crazy but a queue-rewriter would be nontrivial. For now I'm okay
> with saying "add capacity until compaction can mostly keep up." (Most
> people's problem is making compaction LESS aggressive, hence
> https://issues.apache.org/jira/browse/CASSANDRA-2156.)
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Adjusting compaction and memtable settings is a tuning thing. Tuning
is not usually a game changer. Every once in a while you hit a
something wonderful and get 20% or 30% enhancement. But normally you
are in the gain 1%-3% range.
If you are seeing 600 pending compaction tasks regularly you almost
definitely need more hardware.
Re: compaction strategy
Posted by Peter Schuller <pe...@infidyne.com>.
> It does not really make sense to me to go through all these minor merges
> when a full compaction will do a much faster and better job.
In a system heavily reliant on caching (platter drives, large data
sizes, much larger than RAM) major compactions can be very detrimental
to performance due to the effects of the temporary spike in data size
and cache coldness. Sounds like it makes good sense in your situation
though.
--
/ Peter Schuller
Re: compaction strategy
Posted by Terje Marthinussen <tm...@gmail.com>.
Sorry, I was referring to the claim that "one big file" was a problem, not
the non-overlapping part.
If you never compact to a single file, you never get rid of all
generations/duplicates.
With non-overlapping files covering small enough token ranges, compacting
down to one file is not a big issue.
Terje
On Mon, May 9, 2011 at 8:52 PM, David Boxenhorn <da...@taotown.com> wrote:
> If they each have their own copy of the data, then they are *not*
> non-overlapping!
>
> If you have non-overlapping SSTables (and you know the min/max keys), it's
> like having one big SSTable because you know exactly where each row is, and
> it becomes easy to merge a new SSTable in small batches, rather than in one
> huge batch.
>
> The only step that you have to add to the current merge process is, when
> you going to write a new SSTable, if it's too big, to write N
> (non-overlapping!) pieces instead.
>
>
> On Mon, May 9, 2011 at 12:46 PM, Terje Marthinussen <
> tmarthinussen@gmail.com> wrote:
>
>> Yes, agreed.
>>
>> I actually think cassandra has to.
>>
>> And if you do not go down to that single file, how do you avoid getting
>> into a situation where you can very realistically end up with 4-5 big
>> sstables each having its own copy of the same data massively increasing disk
>> requirements?
>>
>> Terje
>>
>> On Mon, May 9, 2011 at 5:58 PM, David Boxenhorn <da...@taotown.com>wrote:
>>
>>> "I'm also not too much in favor of triggering major compactions, because
>>> it mostly have a nasty effect (create one huge sstable)."
>>>
>>> If that is the case, why can't major compactions create many,
>>> non-overlapping SSTables?
>>>
>>> In general, it seems to me that non-overlapping SSTables have all the
>>> advantages of big SSTables (i.e. you know exactly where the data is) without
>>> the disadvantages that come with being big. Why doesn't Cassandra take
>>> advantage of that in a major way?
>>>
>>
>>
>
Re: compaction strategy
Posted by David Boxenhorn <da...@taotown.com>.
If they each have their own copy of the data, then they are *not*
non-overlapping!
If you have non-overlapping SSTables (and you know the min/max keys), it's
like having one big SSTable because you know exactly where each row is, and
it becomes easy to merge a new SSTable in small batches, rather than in one
huge batch.
The only step that you have to add to the current merge process is, when you
going to write a new SSTable, if it's too big, to write N (non-overlapping!)
pieces instead.
On Mon, May 9, 2011 at 12:46 PM, Terje Marthinussen <tmarthinussen@gmail.com
> wrote:
> Yes, agreed.
>
> I actually think cassandra has to.
>
> And if you do not go down to that single file, how do you avoid getting
> into a situation where you can very realistically end up with 4-5 big
> sstables each having its own copy of the same data massively increasing disk
> requirements?
>
> Terje
>
> On Mon, May 9, 2011 at 5:58 PM, David Boxenhorn <da...@taotown.com> wrote:
>
>> "I'm also not too much in favor of triggering major compactions, because
>> it mostly have a nasty effect (create one huge sstable)."
>>
>> If that is the case, why can't major compactions create many,
>> non-overlapping SSTables?
>>
>> In general, it seems to me that non-overlapping SSTables have all the
>> advantages of big SSTables (i.e. you know exactly where the data is) without
>> the disadvantages that come with being big. Why doesn't Cassandra take
>> advantage of that in a major way?
>>
>
>
Re: compaction strategy
Posted by Terje Marthinussen <tm...@gmail.com>.
Yes, agreed.
I actually think cassandra has to.
And if you do not go down to that single file, how do you avoid getting into
a situation where you can very realistically end up with 4-5 big sstables
each having its own copy of the same data massively increasing disk
requirements?
Terje
On Mon, May 9, 2011 at 5:58 PM, David Boxenhorn <da...@taotown.com> wrote:
> "I'm also not too much in favor of triggering major compactions, because it
> mostly have a nasty effect (create one huge sstable)."
>
> If that is the case, why can't major compactions create many,
> non-overlapping SSTables?
>
> In general, it seems to me that non-overlapping SSTables have all the
> advantages of big SSTables (i.e. you know exactly where the data is) without
> the disadvantages that come with being big. Why doesn't Cassandra take
> advantage of that in a major way?
>
Re: compaction strategy
Posted by David Boxenhorn <da...@taotown.com>.
"I'm also not too much in favor of triggering major compactions, because it
mostly have a nasty effect (create one huge sstable)."
If that is the case, why can't major compactions create many,
non-overlapping SSTables?
In general, it seems to me that non-overlapping SSTables have all the
advantages of big SSTables (i.e. you know exactly where the data is) without
the disadvantages that come with being big. Why doesn't Cassandra take
advantage of that in a major way?
Re: compaction strategy
Posted by Jonathan Ellis <jb...@gmail.com>.
You are of course free to reduce the min per bucket to 2.
The fundamental idea of sstables + compaction is to trade disk space
for higher write performance. For most applications this is the right
trade to make on modern hardware... I don't think you'll get very far
trying to get the 2nd without the 1st.
On Wed, May 11, 2011 at 3:49 AM, Terje Marthinussen
<tm...@gmail.com> wrote:
>>
>> Not sure I follow you. 4 sstables is the minimum compaction look for
>> (by default).
>> If there is 30 sstables of ~20MB sitting there because compaction is
>> behind, you
>> will compact those 30 sstables together (unless there is not enough space
>> for
>> that and considering you haven't changed the max compaction threshold (32
>> by
>> default)). And you can increase max threshold.
>> Don't get me wrong, I'm not pretending this works better than it does, but
>> let's not pretend either that it's worth than it is.
>>
>
> Sorry, I am not trying to pretend anything or blow it out of proportions.
> Just reacting to what I see.
> This is what I see after some stress testing of some pretty decent HW.
> 81 Up Normal 181.6 GB 8.33% Token(bytes[30])
>
> 82 Up Normal 501.43 GB 8.33% Token(bytes[313230])
>
> 83 Up Normal 248.07 GB 8.33% Token(bytes[313437])
>
> 84 Up Normal 349.64 GB 8.33% Token(bytes[313836])
>
> 85 Up Normal 511.55 GB 8.33% Token(bytes[323336])
>
> 86 Up Normal 654.93 GB 8.33% Token(bytes[333234])
>
> 87 Up Normal 534.77 GB 8.33% Token(bytes[333939])
>
> 88 Up Normal 525.88 GB 8.33% Token(bytes[343739])
>
> 89 Up Normal 476.6 GB 8.33% Token(bytes[353730])
>
> 90 Up Normal 424.89 GB 8.33% Token(bytes[363635])
>
> 91 Up Normal 338.14 GB 8.33% Token(bytes[383036])
>
> 92 Up Normal 546.95 GB 8.33% Token(bytes[6a])
> .81 has been exposed to a full compaction. It had ~370GB before that and the
> resulting sstable is 165GB.
> The other nodes has only been doing minor compactions
> I think this is a problem.
> You are of course free to disagree.
> I do however recommend doing a simulation on potential worst case scenarios
> if many of the buckets end up with 3 sstables and don't compact for a while.
> The disk space requirements get pretty bad even without getting into
> theoretical worst cases.
> Regards,
> Terje
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: compaction strategy
Posted by Terje Marthinussen <tm...@gmail.com>.
>
>
> Not sure I follow you. 4 sstables is the minimum compaction look for
> (by default).
> If there is 30 sstables of ~20MB sitting there because compaction is
> behind, you
> will compact those 30 sstables together (unless there is not enough space
> for
> that and considering you haven't changed the max compaction threshold (32
> by
> default)). And you can increase max threshold.
> Don't get me wrong, I'm not pretending this works better than it does, but
> let's not pretend either that it's worth than it is.
>
>
Sorry, I am not trying to pretend anything or blow it out of proportions.
Just reacting to what I see.
This is what I see after some stress testing of some pretty decent HW.
81 Up Normal 181.6 GB 8.33% Token(bytes[30])
82 Up Normal 501.43 GB 8.33% Token(bytes[313230])
83 Up Normal 248.07 GB 8.33% Token(bytes[313437])
84 Up Normal 349.64 GB 8.33% Token(bytes[313836])
85 Up Normal 511.55 GB 8.33% Token(bytes[323336])
86 Up Normal 654.93 GB 8.33% Token(bytes[333234])
87 Up Normal 534.77 GB 8.33% Token(bytes[333939])
88 Up Normal 525.88 GB 8.33% Token(bytes[343739])
89 Up Normal 476.6 GB 8.33% Token(bytes[353730])
90 Up Normal 424.89 GB 8.33% Token(bytes[363635])
91 Up Normal 338.14 GB 8.33% Token(bytes[383036])
92 Up Normal 546.95 GB 8.33% Token(bytes[6a])
.81 has been exposed to a full compaction. It had ~370GB before that and the
resulting sstable is 165GB.
The other nodes has only been doing minor compactions
I think this is a problem.
You are of course free to disagree.
I do however recommend doing a simulation on potential worst case scenarios
if many of the buckets end up with 3 sstables and don't compact for a while.
The disk space requirements get pretty bad even without getting into
theoretical worst cases.
Regards,
Terje
Re: compaction strategy
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Tue, May 10, 2011 at 6:20 PM, Terje Marthinussen
<tm...@gmail.com> wrote:
>
>> Everyone may be well aware of that, but I'll still remark that a minor
>> compaction
>> will try to merge "as many 20MB sstables as it can" up to the max
>> compaction
>> threshold (which is configurable). So if you do accumulate some newly
>> created
>> sstable at some point in time, the next minor compaction will take all of
>> them
>> and thus not create a 40 MB sstable, then 80MB etc... Sure there will be
>> more
>> step than with a major compaction, but let's keep in mind we don't
>> merge sstables
>> 2 by 2.
>
> Well, you do kind of merge them 2 by 2 as you look for at least 4 at a time
> ;)
> But yes, 20MB should become at least 80MB. Still quite a few hops to reach
> 100GB.
Not sure I follow you. 4 sstables is the minimum compaction look for
(by default).
If there is 30 sstables of ~20MB sitting there because compaction is behind, you
will compact those 30 sstables together (unless there is not enough space for
that and considering you haven't changed the max compaction threshold (32 by
default)). And you can increase max threshold.
Don't get me wrong, I'm not pretending this works better than it does, but
let's not pretend either that it's worth than it is.
>
>> I'm also not too much in favor of triggering major compactions,
>> because it mostly
>> have a nasty effect (create one huge sstable). Now maybe we could expose
>> the
>> difference factor for which we'll consider sstables in the same bucket
>
> The nasty side effect I am scared of is disk space and to keep the disk
> space under control, I need to get down to 1 file.
>
> As an example:
> 2 days ago, I looked at a system that had gone idle from compaction with
> something like 24 sstables.
> Disk use was 370GB.
>
> After manually triggering full compaction, I was left with a single sstable
> which is 164 GB large.
>
> This means I may need more than 3x the full dataset to survive if certain
> nasty events such as repairs or anti compactions should occur.
> Way more than the recommended 2x.
>
> In the same system, I see nodes reaching up towards 900GB during compaction
> and 5-600GB otherwise.
> This is with OPP, so distribution is not 100% perfect, but I expect these
> 5-600GB nodes to compact down to the <200GB area if a full compaction is
> triggered.
>
> That is way way beyond the recommendation to have 2x the disk space.
>
> You may disagree, but I think this is a problem.
I absolutely do not disagree. I was just arguing that I'm not sure triggering a
major compaction based on some fuzzy heuristic is a good solution to the
problem.
And we do know that compaction could and should be improved, both to
make it have less impact on read when it's behind:
https://issues.apache.org/jira/browse/CASSANDRA-2498
to allow for easily testing different strategy:
https://issues.apache.org/jira/browse/CASSANDRA-1610
as well as redesigning the mechanism:
https://issues.apache.org/jira/browse/CASSANDRA-1608
You'll see in particular in that last ticket comments that segmenting on token
space has been suggested already and there is probably a handful of thread
about vnodes in the mailing list archives.
And I personally think that yes, partitioning the sstables is a good idea.
> Either we need to recommend 3-5x the best case disk usage or we need to fix
> cassandra.
>
> A simple improvement initially may be to change the bucketing strategy if
> you cannot find suitable candidates.
> I believe lucene for instance has a strategy where it can mix a set of small
> index fragments with one large.
> This may be possible to consider as a fallback strategy and just let
> cassandra compact down to 1 file whenever it can.
>
> Ultimately, I think segmenting on token space is the only way to fix this.
> That segmentation could be done by building histograms of your token
> distribution as you compact and the compaction can further adjust the
> segments accordingly as full compactions take place.
>
> This would seem simpler to do than a full vnode based infrastructure.
>
> Terje
>
Re: compaction strategy
Posted by Terje Marthinussen <tm...@gmail.com>.
> Everyone may be well aware of that, but I'll still remark that a minor
> compaction
> will try to merge "as many 20MB sstables as it can" up to the max
> compaction
> threshold (which is configurable). So if you do accumulate some newly
> created
> sstable at some point in time, the next minor compaction will take all of
> them
> and thus not create a 40 MB sstable, then 80MB etc... Sure there will be
> more
> step than with a major compaction, but let's keep in mind we don't
> merge sstables
> 2 by 2.
>
Well, you do kind of merge them 2 by 2 as you look for at least 4 at a time
;)
But yes, 20MB should become at least 80MB. Still quite a few hops to reach
100GB.
I'm also not too much in favor of triggering major compactions,
> because it mostly
> have a nasty effect (create one huge sstable). Now maybe we could expose
> the
> difference factor for which we'll consider sstables in the same bucket
>
The nasty side effect I am scared of is disk space and to keep the disk
space under control, I need to get down to 1 file.
As an example:
2 days ago, I looked at a system that had gone idle from compaction with
something like 24 sstables.
Disk use was 370GB.
After manually triggering full compaction, I was left with a single sstable
which is 164 GB large.
This means I may need more than 3x the full dataset to survive if certain
nasty events such as repairs or anti compactions should occur.
Way more than the recommended 2x.
In the same system, I see nodes reaching up towards 900GB during compaction
and 5-600GB otherwise.
This is with OPP, so distribution is not 100% perfect, but I expect these
5-600GB nodes to compact down to the <200GB area if a full compaction is
triggered.
That is way way beyond the recommendation to have 2x the disk space.
You may disagree, but I think this is a problem.
Either we need to recommend 3-5x the best case disk usage or we need to fix
cassandra.
A simple improvement initially may be to change the bucketing strategy if
you cannot find suitable candidates.
I believe lucene for instance has a strategy where it can mix a set of small
index fragments with one large.
This may be possible to consider as a fallback strategy and just let
cassandra compact down to 1 file whenever it can.
Ultimately, I think segmenting on token space is the only way to fix this.
That segmentation could be done by building histograms of your token
distribution as you compact and the compaction can further adjust the
segments accordingly as full compactions take place.
This would seem simpler to do than a full vnode based infrastructure.
Terje
Re: compaction strategy
Posted by Sylvain Lebresne <sy...@datastax.com>.
On Sat, May 7, 2011 at 7:20 PM, Terje Marthinussen
<tm...@gmail.com> wrote:
> This is an all ssd system. I have no problems with read/write performance
> due to I/O.
> I do have a potential with the crazy explosion you can get in terms of disk
> use if compaction cannot keep up.
>
> As things falls behind and you get many generations of data, yes, read
> performance gets a problem due to the number of sstables.
>
> As things start falling behind, you have a bunch of minor compactions trying
> to merge 20MB (sstables cassandra generally dumps with current config when
> under pressure) into 40 MB into 80MB into....
Everyone may be well aware of that, but I'll still remark that a minor
compaction
will try to merge "as many 20MB sstables as it can" up to the max compaction
threshold (which is configurable). So if you do accumulate some newly created
sstable at some point in time, the next minor compaction will take all of them
and thus not create a 40 MB sstable, then 80MB etc... Sure there will be more
step than with a major compaction, but let's keep in mind we don't
merge sstables
2 by 2.
I'm also not too much in favor of triggering major compactions,
because it mostly
have a nasty effect (create one huge sstable). Now maybe we could expose the
difference factor for which we'll consider sstables in the same bucket
(i.e, of similar
size). As a side note, I think that
https://issues.apache.org/jira/browse/CASSANDRA-1610,
if done correctly, could help in such situation in that one could try
a strategy adapted
to it's work load.
>
> Anyone wants to do the math on how many times you are rewriting the data
> going this route?
>
> There is just no way this can keep up. It will just fall more and more
> behind.
> Only way to recover as I can see would be to trigger a full compaction?
>
> It does not really make sense to me to go through all these minor merges
> when a full compaction will do a much faster and better job.
>
> Terje
>
> On Sat, May 7, 2011 at 9:54 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> On Sat, May 7, 2011 at 2:01 AM, Terje Marthinussen
>> <tm...@gmail.com> wrote:
>> > 1. Would it make sense to make full compactions occur a bit more
>> > aggressive.
>>
>> I'd rather reduce the performance impact of being behind, than do more
>> full compactions: https://issues.apache.org/jira/browse/CASSANDRA-2498
>>
>> > 2. I
>> > would think the code should be smart enough to either trigger a full
>> > compaction and scrap the current queue, or at least merge some of those
>> > pending tasks into larger ones
>>
>> Not crazy but a queue-rewriter would be nontrivial. For now I'm okay
>> with saying "add capacity until compaction can mostly keep up." (Most
>> people's problem is making compaction LESS aggressive, hence
>> https://issues.apache.org/jira/browse/CASSANDRA-2156.)
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
Re: compaction strategy
Posted by Terje Marthinussen <tm...@gmail.com>.
This is an all ssd system. I have no problems with read/write performance
due to I/O.
I do have a potential with the crazy explosion you can get in terms of disk
use if compaction cannot keep up.
As things falls behind and you get many generations of data, yes, read
performance gets a problem due to the number of sstables.
As things start falling behind, you have a bunch of minor compactions trying
to merge 20MB (sstables cassandra generally dumps with current config when
under pressure) into 40 MB into 80MB into....
Anyone wants to do the math on how many times you are rewriting the data
going this route?
There is just no way this can keep up. It will just fall more and more
behind.
Only way to recover as I can see would be to trigger a full compaction?
It does not really make sense to me to go through all these minor merges
when a full compaction will do a much faster and better job.
Terje
On Sat, May 7, 2011 at 9:54 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> On Sat, May 7, 2011 at 2:01 AM, Terje Marthinussen
> <tm...@gmail.com> wrote:
> > 1. Would it make sense to make full compactions occur a bit more
> aggressive.
>
> I'd rather reduce the performance impact of being behind, than do more
> full compactions: https://issues.apache.org/jira/browse/CASSANDRA-2498
>
> > 2. I
> > would think the code should be smart enough to either trigger a full
> > compaction and scrap the current queue, or at least merge some of those
> > pending tasks into larger ones
>
> Not crazy but a queue-rewriter would be nontrivial. For now I'm okay
> with saying "add capacity until compaction can mostly keep up." (Most
> people's problem is making compaction LESS aggressive, hence
> https://issues.apache.org/jira/browse/CASSANDRA-2156.)
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: compaction strategy
Posted by Jonathan Ellis <jb...@gmail.com>.
On Sat, May 7, 2011 at 2:01 AM, Terje Marthinussen
<tm...@gmail.com> wrote:
> 1. Would it make sense to make full compactions occur a bit more aggressive.
I'd rather reduce the performance impact of being behind, than do more
full compactions: https://issues.apache.org/jira/browse/CASSANDRA-2498
> 2. I
> would think the code should be smart enough to either trigger a full
> compaction and scrap the current queue, or at least merge some of those
> pending tasks into larger ones
Not crazy but a queue-rewriter would be nontrivial. For now I'm okay
with saying "add capacity until compaction can mostly keep up." (Most
people's problem is making compaction LESS aggressive, hence
https://issues.apache.org/jira/browse/CASSANDRA-2156.)
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com