You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Redmumba <re...@gmail.com> on 2014/10/20 19:24:03 UTC

Why does recompacting a table with no changes or tombstones result in work?

Based on lots of good feedback from this list, I moved to a date-based set
of tables, which a unique per day.  However, I am manually compacting
previous days' tables to keep the amount of files required to be opened to
a minimum.

I ran into an interesting issue--when I run compaction on a table that is
already compacted, it still, well... compacts.  The table's TTL is set to
0, there are no deletes or other writes to these tables, and I confirmed
(on disk) that there was only a single set of files.  There are no changes
pending in memory for this table, either.

Why is compaction being performed on a table that has no changes?  Are
there other reasons for compaction to be re-run, or does the compact
command to nodetool just blindly do what it's told?

Andrew

Re: Why does recompacting a table with no changes or tombstones result in work?

Posted by Redmumba <re...@gmail.com>.
Much appreciated, Robert.  I was under the (unfounded) impression that the
tombstones and similar were stored in the index or other auxilliary file.
A false assumption, to be sure.

For readers: To support this specific use case, my plan is to have a job
that simply parses out the "sstable ID" from the table's files to see if
there's more than one for a given table.  If there are, it means there are
multiple sstables that can be compacted.

On Mon, Oct 20, 2014 at 12:03 PM, Robert Coli <rc...@eventbrite.com> wrote:

> On Mon, Oct 20, 2014 at 10:24 AM, Redmumba <re...@gmail.com> wrote:
>
>> I ran into an interesting issue--when I run compaction on a table that is
>> already compacted, it still, well... compacts.  The table's TTL is set to
>> 0, there are no deletes or other writes to these tables, and I confirmed
>> (on disk) that there was only a single set of files.  There are no changes
>> pending in memory for this table, either.
>>
>> Why is compaction being performed on a table that has no changes?  Are
>> there other reasons for compaction to be re-run, or does the compact
>> command to nodetool just blindly do what it's told?
>>
>
> The latter. The compaction process doesn't "know" that the file is
> compacted and it doesn't "know" that there have been no writes since it was
> compacted. It also doesn't know that the file doesn't contain TTLed columns
> or deletes with timestamps in the future from write time at first
> compaction, but which a second compaction would turn into tombstones. Or
> tombstones which would be removed after gc_grace_seconds, etc...
>
> In trunk Cassandra, SSTables are marked "repaired" which would prevent
> them from repairing. In theory maybe (?) a similar technique be used with
> regards to compaction, but Cassandra dev team quite rationally does not
> optimize for the case where the dataset is static...
>
> =Rob
>
>

Re: Why does recompacting a table with no changes or tombstones result in work?

Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Oct 20, 2014 at 10:24 AM, Redmumba <re...@gmail.com> wrote:

> I ran into an interesting issue--when I run compaction on a table that is
> already compacted, it still, well... compacts.  The table's TTL is set to
> 0, there are no deletes or other writes to these tables, and I confirmed
> (on disk) that there was only a single set of files.  There are no changes
> pending in memory for this table, either.
>
> Why is compaction being performed on a table that has no changes?  Are
> there other reasons for compaction to be re-run, or does the compact
> command to nodetool just blindly do what it's told?
>

The latter. The compaction process doesn't "know" that the file is
compacted and it doesn't "know" that there have been no writes since it was
compacted. It also doesn't know that the file doesn't contain TTLed columns
or deletes with timestamps in the future from write time at first
compaction, but which a second compaction would turn into tombstones. Or
tombstones which would be removed after gc_grace_seconds, etc...

In trunk Cassandra, SSTables are marked "repaired" which would prevent them
from repairing. In theory maybe (?) a similar technique be used with
regards to compaction, but Cassandra dev team quite rationally does not
optimize for the case where the dataset is static...

=Rob