You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Yifan Cai (Jira)" <ji...@apache.org> on 2026/09/08 22:02:00 UTC

[jira] [Commented] (CASSANDRA-21666) Compression dictionary training fails with "insufficient samples" on large-chunk tables (SSTableChunkSampler per-SSTable allocation rounds to zero)

    [ https://issues.apache.org/jira/browse/CASSANDRA-21666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113033#comment-18113033 ] 

Yifan Cai commented on CASSANDRA-21666:
---------------------------------------

+1 on the patch, with a nit comment left in the PR.

> Compression dictionary training fails with "insufficient samples" on large-chunk tables (SSTableChunkSampler per-SSTable allocation rounds to zero)
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-21666
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-21666
>             Project: Apache Cassandra
>          Issue Type: Bug
>          Components: Feature/Compression
>            Reporter: Stefan Miklosovic
>            Assignee: Stefan Miklosovic
>            Priority: Normal
>             Fix For: 6.0-alpha3, 7.x
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> h2. Summary
> {{nodetool compressiondictionary train}} fails with {{Insufficient samples for training: N (minimum required: 11)}} on tables that have a large {{chunk_length_in_kb}} and many SSTables, even when there are gigabytes of data available to sample.
> h2. Root cause
> {{SSTableChunkSampler}} ({_}src/java/org/apache/cassandra/db/compression/SSTableChunkSampler.java{_}) allocates the number of chunks to sample from each SSTable using integer division that truncates to zero for most SSTables when the sample target is small.
> {{{}calculateTargetChunkCount(){}}}:
> {code:java}
> int averageChunkSize = totalDataSize / totalChunks;      // ~ chunk_length_in_kb
> return config.maxTotalSampleSize / averageChunkSize;     // target shrinks as chunk size grows
> {code}
> With the default {{maxTotalSampleSize = 8 MiB}} and 64 KiB chunks: {{{}targetChunkCount = 8 MiB / 64 KiB = 128{}}}.
> {{{}sampleChunksFromSSTables(){}}}:
> {code:java}
> long chunksFromThisSSTable = Math.min((targetChunkCount * info.chunkCount) / totalChunks, remainingTarget);
> if (chunksFromThisSSTable <= 0)
>     continue;   // <-- SSTable skipped entirely
> {code}
> {{(targetChunkCount * info.chunkCount) / totalChunks}} is integer division. An SSTable contributes at least one chunk only when {{{}info.chunkCount >= totalChunks / targetChunkCount{}}}. With {{targetChunkCount = 128}} and ~387 SSTables (~100k total chunks), that threshold is ~780 chunks (~50 MB) — so every SSTable below it rounds to 0 and is skipped, leaving ~8 samples, under the 11-sample minimum, and training fails.
> Because {{targetChunkCount}} is 16x larger at 4 KiB chunks (2048), small-chunk tables clear the threshold and are unaffected — which is why this only manifests at larger {{{}chunk_length{}}}.
> ||chunk_length_in_kb||targetChunkCount||samples collected||result||
> |4|2048|~2418|COMPLETED|
> |64|128|8|FAILED (minimum required: 11)|
> h2. Steps to reproduce
>  # Create a table with {{ZstdDictionaryCompressor}} and {{{}chunk_length_in_kb: 64{}}}.
>  # Write several GB so the table has many SSTables (LCS L0, TWCS, or any table not yet fully compacted; ~387 SSTables / ~6.5 GB in this repro).
>  # {{nodetool compressiondictionary train <ks> <table>}}
> *Expected:* the sampler collects a representative sample and a dictionary is trained.
> *Actual:*
> {noformat}
> Status: FAILED | Samples: 8 | Size: 0.22 MiB | Elapsed: 1s
> Training failed for <ks>.<table>
> Reason: Insufficient samples for training: 8 (minimum required: 11)
> {noformat}
> The identical dataset with {{chunk_length_in_kb: 4}} trains successfully (~2418 samples).
> h2. Suggested fix
>  * In {{{}sampleChunksFromSSTables(){}}}, guarantee at least one chunk from each suitable SSTable, bounded by the remaining budget — e.g. {{Math.max(1, (targetChunkCount * info.chunkCount) / totalChunks)}} while still respecting {{{}remainingTarget{}}}/{{{}maxTotalSampleSize{}}}.
>  * And/or floor {{targetChunkCount}} in {{calculateTargetChunkCount()}} at {{max(MIN_SAMPLES, sstableCount)}} so it can never fall below what is needed to reach the minimum.
>  * Separately, either make {{--force}} actually bypass the sample-count minimum, or correct the message that recommends it.
> h2. Environment
> Cassandra 6.0-alpha-SNAPSHOT (trunk), single node.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org