C* 3.5 - Not all SSTables were removed in DTCS

[Jacek Luczak <di...@gmail.com>]

Fellow C* users,

I've got a cluster of C* 3.5 serving a single keyspace with DTCS table
and no deletes. We knew that data does not expire on time (even after
gc_grace_period) - that's sth we wanted to investigate later and
eventually let C* to keep it longer. The moment when C* decided to
remove old data happened yesterday and on a one node we've reduced the
data size by 40% (removing ~350 SSTables). On remaining nodes that was
only ~5%. That's of course wrong and all nodes should get down by 40%.
A closer look and we really found that ~40% is out-of-TTL and that
maps to ~350 SSTables.

I've triggered a user defined compaction on few of the tables but ...
nodetool compact

Re: C* 3.5 - Not all SSTables were removed in DTCS

[Jacek Luczak <di...@gmail.com>]

/* Sorry for the mess, accidental Ctrl-Enter */

Fellow C* users,

I've got a cluster of C* 3.5 serving a single keyspace with DTCS table
and no deletes. We knew that data does not expire on time (even after
gc_grace_period) - that's sth we wanted to investigate later and
eventually let C* to keep it longer. The moment when C* decided to
remove old data happened yesterday and on a one node we've reduced the
data size by 40% (removing ~350 SSTables). On remaining nodes that was
only ~5%. That's of course wrong and all nodes should get down by 40%.
A closer look and we really found that ~40% is out-of-TTL and that
maps to ~350 SSTables.

I've triggered a user defined compaction on few of the SStables but ...
nodetool compact exited with 0 a second after and file remain
untouched (no entry in the debug.log about compaction). The files are
mmap()ed in the process as well.

Is there a known bug for this behaviour?

Why one node removed *all* data at once (there's data that should
expire 3 months ago) while others not?

All nodes are equal in spec and configuration.

-Jacek


2016-12-08 14:11 GMT+01:00 Jacek Luczak <di...@gmail.com>:
> Fellow C* users,
>
> I've got a cluster of C* 3.5 serving a single keyspace with DTCS table
> and no deletes. We knew that data does not expire on time (even after
> gc_grace_period) - that's sth we wanted to investigate later and
> eventually let C* to keep it longer. The moment when C* decided to
> remove old data happened yesterday and on a one node we've reduced the
> data size by 40% (removing ~350 SSTables). On remaining nodes that was
> only ~5%. That's of course wrong and all nodes should get down by 40%.
> A closer look and we really found that ~40% is out-of-TTL and that
> maps to ~350 SSTables.
>
> I've triggered a user defined compaction on few of the tables but ...
> nodetool compact