You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by "smiklosovic (via GitHub)" <gi...@apache.org> on 2026/09/08 12:00:27 UTC
[PR] CASSANDRA-21666 Fix compression dictionary training failing with "insufficient samples" on large-chunk tables [cassandra]
smiklosovic opened a new pull request, #5131:
URL: https://github.com/apache/cassandra/pull/5131
Thanks for sending a pull request! Here are some tips if you're new here:
* Ensure you have added or run the [appropriate tests](https://cassandra.apache.org/_/development/testing.html) for your PR.
* Be sure to keep the PR description updated to reflect all changes.
* Write your PR title to summarize what this PR proposes.
* If possible, provide a concise example to reproduce the issue for a faster review.
* Read our [contributor guidelines](https://cassandra.apache.org/_/development/index.html)
* If you're making a documentation change, see our [guide to documentation contribution](https://cassandra.apache.org/_/development/documentation.html)
Commit messages should follow the following format:
```
<One sentence description, usually Jira title or CHANGES.txt summary>
<Optional lengthier description (context on patch)>
patch by <Authors>; reviewed by <Reviewers> for CASSANDRA-#####
Co-authored-by: Name1 <email1>
Co-authored-by: Name2 <email2>
```
The [Cassandra Jira](https://issues.apache.org/jira/projects/CASSANDRA/issues/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21666 Fix compression dictionary training failing with "insufficient samples" on large-chunk tables [cassandra]
Posted by "smiklosovic (via GitHub)" <gi...@apache.org>.
smiklosovic closed pull request #5131: CASSANDRA-21666 Fix compression dictionary training failing with "insufficient samples" on large-chunk tables
URL: https://github.com/apache/cassandra/pull/5131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org
Re: [PR] CASSANDRA-21666 Fix compression dictionary training failing with "insufficient samples" on large-chunk tables [cassandra]
Posted by "yifan-c (via GitHub)" <gi...@apache.org>.
yifan-c commented on code in PR #5131:
URL: https://github.com/apache/cassandra/pull/5131#discussion_r3960629933
##########
src/java/org/apache/cassandra/db/compression/SSTableChunkSampler.java:
##########
@@ -214,12 +214,22 @@ static SamplingStats sampleChunksFromSSTables(List<SSTableChunkInfo> sstableInfo
break;
}
- // Calculate how many chunks to sample from this SSTable (proportional to its size)
+ // Calculate how many chunks to sample from this SSTable (proportional to its size), but always at
+ // least one chunk per SSTable, bounded by the remaining budget. The proportional share is integer
+ // division and truncates to ZERO once there are more SSTables than the target chunk count. That
+ // happens with large chunk sizes, because targetChunkCount = maxTotalSampleSize / chunkSize shrinks
+ // as chunks grow (e.g. 8 MiB / 64 KiB = 128): spread across hundreds of SSTables the per-SSTable
+ // share rounds to zero, so most SSTables are skipped and far fewer samples are collected than
+ // intended — often below the trainer's minimum, failing training despite gigabytes of data. The
+ // Math.max(1, ...) floor guarantees every SSTable contributes; the maxTotalSampleSize byte budget
+ // (via remainingTarget) still caps the total that is actually sampled.
long remainingTarget = Math.min(targetChunkCount - sampleCount, (config.maxTotalSampleSize - totalSampleSize) / info.chunkSize);
- long chunksFromThisSSTable = Math.min((targetChunkCount * info.chunkCount) / totalChunks, remainingTarget);
+ long proportionalChunks = (targetChunkCount * info.chunkCount) / totalChunks;
+ long chunksFromThisSSTable = Math.min(Math.max(1, proportionalChunks), remainingTarget);
Review Comment:
nit: Since it now greedily picks at least 1 chunk based on the list order, how about shuffling the `sstableInfos` list to preserve the randomness?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org