You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (Jira)" <ji...@apache.org> on 2026/08/26 16:32:00 UTC

[jira] [Updated] (CASSANDRA-21615) UCS never compacts tiny high-density SSTables streamed by repair

     [ https://issues.apache.org/jira/browse/CASSANDRA-21615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brandon Williams updated CASSANDRA-21615:
-----------------------------------------
     Bug Category: Parent values: Degradation(12984)Level 1 values: Resource Management(12995)
       Complexity: Normal
    Discovered By: User Report
    Fix Version/s: 5.0.x
                   6.0.x
                   7.x
         Severity: Normal
           Status: Open  (was: Triage Needed)

> UCS never compacts tiny high-density SSTables streamed by repair
> ----------------------------------------------------------------
>
>                 Key: CASSANDRA-21615
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-21615
>             Project: Apache Cassandra
>          Issue Type: Bug
>          Components: Local/Compaction/UCS
>            Reporter: Hayato Shimizu
>            Priority: Normal
>             Fix For: 5.0.x, 6.0.x, 7.x
>
>
> Cassandra Version: 5.0.8
> This was observed on a production cluster, and it took a while to rule out misconfiguration before it became clear that UCS has no configuration that will compact these repair-streamed SSTables.
> On tables that receive frequent repair streams, UCS accumulates a very large number of tiny SSTables that never get compacted. The live SSTable count climbs into the thousands per node while total disk usage barely increases, because each streamed file is only hundreds of bytes to a few KB.
> We could see in the metrics the SSTable count increased only during repair activity and did not decrease afterwards, and pending compactions stayed near zero throughout. So this was definitely not a compaction backlog. UCS did not appear to be selecting these SSTables for compaction in the first place.
> The following was seen on a production node, via {{{}sstablemetadata{}}}:
>  * a file of a few hundred bytes, local token space coverage on the order of 1e-11, so density (size ÷ coverage) on the order of 1e13
>  * a file of a few hundred bytes, coverage on the order of 1e-10, density on the order of 1e12
>  * a file of a few thousand bytes, coverage on the order of 1e-11, density on the order of 1e14
> At these densities each file was assigned to a very high level, far above the table's normal SSTables. Each covered such a narrow token range that it intersected no other SSTable in its level, so no overlap set of size >= 2 ever formed and the threshold was never met.
> Lowering the threshold was the obvious first thing to try, but unfortunately it did not help. Setting scaling_parameters to 'T4, T4, N' (threshold 2 in the upper levels), and separately updated min_sstable_size to 0MiB, then only a handful of files whose ranges happened to intersect compacted, so the table stayed at a very large SSTable count. This makes sense since a threshold of 2 does nothing when nothing intersects.
> overlap_inclusion_method was on its default of TRANSITIVE throughout, so transitive inclusion was already in effect and did not pull these files in. As far as we can tell that's because they are denser than the table's larger SSTables, so a lower-level compaction never seems to include them, though we have not confirmed this in the code.
> Checked JIRA before filing and didn't find this — CASSANDRA-16552 and 16638 are the closest but they're about the anticompaction race under incremental repair, which is different (this is fully unrepaired, no anticompaction).
> We'd like to know whether the above explanation is correct, whether this is a bug or a known limitation of UCS, and whether there's a recommended way to handle repair-streamed SSTables like these on UCS tables in the meantime.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org