what are ways to keep the SSTable Count down low

[Dathan Pattishall <da...@gmail.com>]

How do you set the compaction threshold from storage-conf.xml? is this possible?

What is the consensus on a basic Key-Value store of setting the
compactionthreshold min/max from

./nodetool --host=localhost --port=8181 getcompactionthreshold
Current compaction threshold: Min=4, Max=32

to say something like 2/5 ?


The next question is why do you want to change it?

I noticed that all my outages are due to compaction. The SSTable count
grows from 4-11 and then a compaction happens taking that server
offline even with all the threshold tricks in cassandra.in.sh
(-Dcassandra.compaction.priority=1). So, playing with this value seems
to me to be the right course of action.

Re: what are ways to keep the SSTable Count down low

[Jonathan Ellis <jb...@gmail.com>]

On Mon, Sep 20, 2010 at 4:57 PM, Peter Schuller
<pe...@infidyne.com> wrote:
>> Not in 0.6 I believe, but I believe this recently went into trunk for
>> the upcoming 0.7.
>
> By "not in 0.6" i mean in terms of the configuration.

This is ready for 0.6.6, fwiw.

-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: what are ways to keep the SSTable Count down low

[Peter Schuller <pe...@infidyne.com>]

> Not in 0.6 I believe, but I believe this recently went into trunk for
> the upcoming 0.7.

By "not in 0.6" i mean in terms of the configuration. If you are
willing to modify the source you can change it fairly easily by
changing the private constants towards the beginning of
org/apache/cassandra/db/CompactionManager.java:

    private int minimumCompactionThreshold = 4; // compact this many
sstables min at a time
    private int maximumCompactionThreshold = 32; // compact this many
sstables max at a time

Note that increasing the min will tend to mean more overall
compaction, while decreasing it means less overall compaction. The
trade-off is potentially slower reads due to more sstables having to
be consulted (meaning, more seeking).

If you're worried about decreasing amount of compaction, you don't
want to decrease the 'max'. It *might*, under some circumstances,
contribute to decreasing the impact of compaction once it happens, but
that would be highly dependent on I/O scheduling and other
circumstances. I wouldn't touch if unless I knew specifically there
was a reason to.

-- 
/ Peter Schuller

Re: what are ways to keep the SSTable Count down low

[Peter Schuller <pe...@infidyne.com>]

> How do you set the compaction threshold from storage-conf.xml? is this possible?

Not in 0.6 I believe, but I believe this recently went into trunk for
the upcoming 0.7.

> What is the consensus on a basic Key-Value store of setting the
> compactionthreshold min/max from
>
> ./nodetool --host=localhost --port=8181 getcompactionthreshold
> Current compaction threshold: Min=4, Max=32
>
> to say something like 2/5 ?

I make no claims about consensus, but in general if compaction is
affecting your node so adversely that it's effectively going offline
for the duration, while tweaking those settings may affect e.g.
frequency of compactions, it won't really fundamentally mitigate their
effects when they do happen.

I am curious about and other cases where people have sever problems
with compaction. In particular, what kind of row/column sizes are we
talking about? I would expect compaction to be a bigger killer for
data sets where I/O becomes the bottleneck rather than CPU (which
would tend to mean larger rows with fewer columns, meaning less CPU
bound compaction). It would be interest to confirm whether this is the
case.

There is some ongoing work to mitigate the impact of compaction and it
would be interesting to gather reports about the nature of the
problems people are having with it.

Have you observed 'iostat -x 1' or similar during compaction? Is the
underlying storage maxed out? What kind of I/O load do you normally
see due to regular traffic when compaction is not happening?

-- 
/ Peter Schuller

Re: what are ways to keep the SSTable Count down low

[Edward Capriolo <ed...@gmail.com>]

On Mon, Sep 20, 2010 at 3:14 PM, Dathan Pattishall <da...@gmail.com> wrote:
> How do you set the compaction threshold from storage-conf.xml? is this possible?
>
> What is the consensus on a basic Key-Value store of setting the
> compactionthreshold min/max from
>
> ./nodetool --host=localhost --port=8181 getcompactionthreshold
> Current compaction threshold: Min=4, Max=32
>
> to say something like 2/5 ?
>
>
> The next question is why do you want to change it?
>
> I noticed that all my outages are due to compaction. The SSTable count
> grows from 4-11 and then a compaction happens taking that server
> offline even with all the threshold tricks in cassandra.in.sh
> (-Dcassandra.compaction.priority=1). So, playing with this value seems
> to me to be the right course of action.
>

You should look at your memtable settings memtable flushes happen
because of either
1) time
2) size
3) activity
Why are they happening for you?

If you raise your memtable settings you will get larger memtables.
This results in larger sstables that will compact less often.

Lowering the compaction probably will not do what you want because
this will result in the same data being compacted again and again.
Remember the goal of minor compaction is to combine X like sized
SSTables.

If your nodes are going out during compaction they are likely
overtaxed. I had great luck with the compaction priority. 6.5 features
'dynamic snitch' which should help lead traffic away from compacting
nodes. You may want to give it a try.