You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Kevin Burton <bu...@spinn3r.com> on 2014/07/24 23:56:27 UTC
2x disk space required for full compaction? Don't vnodes help this problem?
I just bootstrapped a new node.
The box had about 220GB of data on it on a 400GB SSD drive.
I triggered a full compaction after it bootstrapped, and it ran out of disk
space about 15 minutes later. so now that node is dead :-(
I would have assumed that vnodes meant that I could keep my drive near 100%
full…
so during a major compaction it would just compact the first vnode, then
move on to the second.
this would be analogous to bigtable / hbase regions.
but … that doesn't seem to be the case. (so bad assumption on my part)
Both in terms of me actually seeing the disk fill up, and also the case of
my disk not having separate SSTables for each vnode.
So now I have these SSDs that I have to keep at > 50% capacity at all times.
I can see why on HDDs having too many files would be an issue.
But on SSDs this is less of a problem.
Perhaps some hybrid where vnodes are chunked together in one contiguous
region?
Is there a way to fix this problem? I would like to get more usage out of
my SSDs...
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: 2x disk space required for full compaction? Don't vnodes help
this problem?
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Jul 24, 2014 at 3:30 PM, Colin Clark <co...@clark.ws> wrote:
> Triggering a major compaction is usually not a good idea.
>
I think this is a little over-stated, there are an assortment of cases in
which even perioidic major compaction is really a-ok. :D
=Rob
Re: 2x disk space required for full compaction? Don't vnodes help
this problem?
Posted by Colin Clark <co...@clark.ws>.
Triggering a major compaction is usually not a good idea.
If you've got ssd's, go leveled as DuyHai says. The results will be tasty.
--
Colin
320-221-9531
On Jul 24, 2014, at 5:28 PM, Kevin Burton <bu...@spinn3r.com> wrote:
This was after a bootstrap… so I triggered a major compaction. Should I
just turn on leveled compaction and then never do a major compaction?
On Thu, Jul 24, 2014 at 3:09 PM, DuyHai Doan <do...@gmail.com> wrote:
> If you're using SizeTieredCompactionStrategy the disk space may double
> temporarily during compaction. That's one of the big drawback of
> SizedTiered. Since you're on SSD, why not test switching to
> LeveledCompaction ? Put a node on write survey mode to see if this change
> has any impact on your I/O, CPU and node stability.
>
>
> On Thu, Jul 24, 2014 at 11:56 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> I just bootstrapped a new node.
>>
>> The box had about 220GB of data on it on a 400GB SSD drive.
>>
>> I triggered a full compaction after it bootstrapped, and it ran out of
>> disk space about 15 minutes later. so now that node is dead :-(
>>
>> I would have assumed that vnodes meant that I could keep my drive near
>> 100% full…
>>
>> so during a major compaction it would just compact the first vnode, then
>> move on to the second.
>>
>> this would be analogous to bigtable / hbase regions.
>>
>> but … that doesn't seem to be the case. (so bad assumption on my part)
>> Both in terms of me actually seeing the disk fill up, and also the case of
>> my disk not having separate SSTables for each vnode.
>>
>> So now I have these SSDs that I have to keep at > 50% capacity at all
>> times.
>>
>> I can see why on HDDs having too many files would be an issue.
>>
>> But on SSDs this is less of a problem.
>>
>> Perhaps some hybrid where vnodes are chunked together in one contiguous
>> region?
>>
>> Is there a way to fix this problem? I would like to get more usage out of
>> my SSDs...
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>> <http://spinn3r.com>
>>
>>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: 2x disk space required for full compaction? Don't vnodes help
this problem?
Posted by Kevin Burton <bu...@spinn3r.com>.
This was after a bootstrap… so I triggered a major compaction. Should I
just turn on leveled compaction and then never do a major compaction?
On Thu, Jul 24, 2014 at 3:09 PM, DuyHai Doan <do...@gmail.com> wrote:
> If you're using SizeTieredCompactionStrategy the disk space may double
> temporarily during compaction. That's one of the big drawback of
> SizedTiered. Since you're on SSD, why not test switching to
> LeveledCompaction ? Put a node on write survey mode to see if this change
> has any impact on your I/O, CPU and node stability.
>
>
> On Thu, Jul 24, 2014 at 11:56 PM, Kevin Burton <bu...@spinn3r.com> wrote:
>
>> I just bootstrapped a new node.
>>
>> The box had about 220GB of data on it on a 400GB SSD drive.
>>
>> I triggered a full compaction after it bootstrapped, and it ran out of
>> disk space about 15 minutes later. so now that node is dead :-(
>>
>> I would have assumed that vnodes meant that I could keep my drive near
>> 100% full…
>>
>> so during a major compaction it would just compact the first vnode, then
>> move on to the second.
>>
>> this would be analogous to bigtable / hbase regions.
>>
>> but … that doesn't seem to be the case. (so bad assumption on my part)
>> Both in terms of me actually seeing the disk fill up, and also the case of
>> my disk not having separate SSTables for each vnode.
>>
>> So now I have these SSDs that I have to keep at > 50% capacity at all
>> times.
>>
>> I can see why on HDDs having too many files would be an issue.
>>
>> But on SSDs this is less of a problem.
>>
>> Perhaps some hybrid where vnodes are chunked together in one contiguous
>> region?
>>
>> Is there a way to fix this problem? I would like to get more usage out of
>> my SSDs...
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>> <http://spinn3r.com>
>>
>>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: 2x disk space required for full compaction? Don't vnodes help
this problem?
Posted by DuyHai Doan <do...@gmail.com>.
If you're using SizeTieredCompactionStrategy the disk space may double
temporarily during compaction. That's one of the big drawback of
SizedTiered. Since you're on SSD, why not test switching to
LeveledCompaction ? Put a node on write survey mode to see if this change
has any impact on your I/O, CPU and node stability.
On Thu, Jul 24, 2014 at 11:56 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> I just bootstrapped a new node.
>
> The box had about 220GB of data on it on a 400GB SSD drive.
>
> I triggered a full compaction after it bootstrapped, and it ran out of
> disk space about 15 minutes later. so now that node is dead :-(
>
> I would have assumed that vnodes meant that I could keep my drive near
> 100% full…
>
> so during a major compaction it would just compact the first vnode, then
> move on to the second.
>
> this would be analogous to bigtable / hbase regions.
>
> but … that doesn't seem to be the case. (so bad assumption on my part)
> Both in terms of me actually seeing the disk fill up, and also the case of
> my disk not having separate SSTables for each vnode.
>
> So now I have these SSDs that I have to keep at > 50% capacity at all
> times.
>
> I can see why on HDDs having too many files would be an issue.
>
> But on SSDs this is less of a problem.
>
> Perhaps some hybrid where vnodes are chunked together in one contiguous
> region?
>
> Is there a way to fix this problem? I would like to get more usage out of
> my SSDs...
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>