Compaction no longer working properly

[Robert Wille <rw...@fold3.com>]

I have a database which has a fair amount of churn. When I need to update a data structure, I create a new one, and when it is complete, I delete the old one. I have gc_grace_seconds=0, so the space for the old data structures should be reclaimed on the next compaction. This has been working fine for months. Unfortunately, compaction has stopped doing its job, and about a week ago disk space started increasing steadily. Disk space usage has increased almost 50% in the last week.

I do see compaction tasks get kicked off, but they complete very quickly, generally in well less than a minute. Compaction of the bigger tables used to take more than an hour. Admittedly, its been quite a while since I’ve watched compactions run, and I probably haven’t watched them since we last upgraded, so perhaps there’s been a change in how LeveledCompaction works. Our last upgrade was well more than a week ago, so an upgrade isn’t responsible for this change in behavior. We’re running 2.0.14 currently.

Looking at the logs, I see stuff like this:

INFO [CompactionExecutor:49180] 2015-07-24 09:31:45,960 CompactionTask.java (line 296) Compacted 2 sstables to [/var/lib/cassandra/data/fold31_browse/node/fold31_browse-node-jb-11110,].  117,066,921 bytes to 117,346,402 (~100% of original) in 33,766ms = 3.314288MB/s.  1,444,907 total partitions merged to 1,400,563.  Partition merge counts were {1:1356219, 2:44344, }

So, compaction seems to be doing something, just not very much. I’ve checked the process that deletes stale data, and it is doing its job. Plenty of deletion going on, just no space reclamation.

Any thoughts?

Thanks in advance

Robert