Size-tiered Compaction runs out of memory

[Redmumba <re...@gmail.com>]

I am having an issue on multiple machines where it's simply filling up the
disk space during what I can only assume is a compaction.  For example, the
average node cluster-wide is around 900GB according to DSE
OpsCenter--however, after coming in after the three day weekend, I noticed
that there were 13 hosts that were down in the cluster.

When I investigated, I see several huge sstables that appeared to be in the
middle of compaction when the host ran out of usable disk space:

81G     auditing-audit-jb-17623-Data.db
> 189G    auditing-audit-jb-19863-Data.db
> 182G    auditing-audit-jb-25298-Data.db
> 196G    auditing-audit-jb-30791-Data.db
> 13G     auditing-audit-jb-31003-Data.db
> 12G     auditing-audit-jb-31341-Data.db
> 12G     auditing-audit-jb-31678-Data.db
> 12G     auditing-audit-jb-32019-Data.db
> 766M    auditing-audit-jb-32039-Data.db
> 791M    auditing-audit-jb-32060-Data.db
> 199M    auditing-audit-jb-32065-Data.db
> 52M     auditing-audit-jb-32066-Data.db
> 175G    auditing-audit-jb-8179-Data.db
>
> *643G    auditing-audit-tmp-jb-31207-Data.db32G
> auditing-audit-tmp-jb-32030-Data.db*
>

>From what I can tell, it is reaching the point where it is trying to
compact the ~190G ones into a combined bigger one, and is failing because
of the lack of disk space.  How do I work around this?

Would adjusting the maximum sstables before a compaction is performed help
this situation?  I am currently using the default values provided by
SizeTieredCompactionStrategy in C* 2.0.6.  Or is there a better option for
a continuous-write operation (with TTLs for dropping off old data)?

Thank you for your help!

Andrew

Re: Size-tiered Compaction runs out of memory

[Robert Coli <rc...@eventbrite.com>]

On Mon, Jul 7, 2014 at 9:52 AM, Redmumba <re...@gmail.com> wrote:

> Would adjusting the maximum sstables before a compaction is performed help
> this situation?  I am currently using the default values provided by
> SizeTieredCompactionStrategy in C* 2.0.6.  Or is there a better option for
> a continuous-write operation (with TTLs for dropping off old data)?
>

(Sorry, just saw this line about the workload.)

Redis?

If you have a high write rate and are discarding 100% of data after a TTL,
perhaps a data-store with immutable data files that reconciles row
fragments on read is not ideal for your use case?

https://issues.apache.org/jira/browse/CASSANDRA-6654

Is probably also worth a read...

=Rob

Re: Size-tiered Compaction runs out of memory

[Robert Coli <rc...@eventbrite.com>]

On Mon, Jul 7, 2014 at 9:52 AM, Redmumba <re...@gmail.com> wrote:

> I am having an issue on multiple machines where it's simply filling up the
> disk space during what I can only assume is a compaction.  For example, the
> average node cluster-wide is around 900GB according to DSE
> OpsCenter--however, after coming in after the three day weekend, I noticed
> that there were 13 hosts that were down in the cluster.
>
> When I investigated, I see several huge sstables that appeared to be in
> the middle of compaction when the host ran out of usable disk space:
>
>>
>> *643G    auditing-audit-tmp-jb-31207-Data.db32G
>> auditing-audit-tmp-jb-32030-Data.db*
>>
>
> From what I can tell, it is reaching the point where it is trying to
> compact the ~190G ones into a combined bigger one, and is failing because
> of the lack of disk space.  How do I work around this?
>

You mostly don't. Having enough headroom to compact is a part of Cassandra
capacity planning.

Are you doing a large amount of UPDATE or overwrite load which would lead
to very-efficient compaction [1]? If not, your problem isn't compaction,
it's that you have too much data per node.

The limited workaround available here is to :

1) stop automatic minor compactions (via the thresholds)
2) run userdefinedcompaction via JMX, combining only enough of the large
files to successfully compact

But 2) is not likely to be very useful unless your compactions are actually
reclaiming space. Probably "just" increase the size of your cluster.

=Rob
[1] grep % /path/to/system.log # are these percentages below 95%? if not,
compaction will not reclaim disk space when done!