Question on pending tasks in compaction manager

[Martin McGovern <ma...@gmail.com>]

Hi All,

Could someone explain why the compaction manager stops compacting when it
has a number of pending tasks?

I have a test cluster that I am using to stress test IO throughput, i.e.
find out what a safe load for our hardware is. Over a 16 hour period my
node cluster completes approximately 49,000 tasks per node. After stopping
my test compaction continues for a few minutes then stops. There are ~7,000
tasks still pending. No more tasks will be executed until I start another
test and the 7000 pending will never be executed.

I'm using leveled compaction with 5MB SS tables and my tests have a 50:50
read:write ratio. Each value is a 10K byte array with random content.

Thanks,
Martin

Re: Question on pending tasks in compaction manager

[Jonathan Ellis <jb...@gmail.com>]

"Pending compactions" is just an estimate of "how many compactions
does Cassandra think it will take to get to fully-compacted state;"
there are no actual tasks enqueued anywhere.

You could enable debug logging on org.apache.cassandra.db.compaction,
and force a compaction with nodetool to see why no compactions happen
when the estimate says there is still work to do.

On Fri, Jun 29, 2012 at 4:27 AM, Martin McGovern
<ma...@gmail.com> wrote:
> Hi All,
>
> Could someone explain why the compaction manager stops compacting when it
> has a number of pending tasks?
>
> I have a test cluster that I am using to stress test IO throughput, i.e.
> find out what a safe load for our hardware is. Over a 16 hour period my node
> cluster completes approximately 49,000 tasks per node. After stopping my
> test compaction continues for a few minutes then stops. There are ~7,000
> tasks still pending. No more tasks will be executed until I start another
> test and the 7000 pending will never be executed.
>
> I'm using leveled compaction with 5MB SS tables and my tests have a 50:50
> read:write ratio. Each value is a 10K byte array with random content.
>
> Thanks,
> Martin



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com