You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stefan Miklosovic (Jira)" <ji...@apache.org> on 2023/09/25 10:47:00 UTC
[jira] [Comment Edited] (CASSANDRA-18773) Compactions are slow
[ https://issues.apache.org/jira/browse/CASSANDRA-18773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17768628#comment-17768628 ]
Stefan Miklosovic edited comment on CASSANDRA-18773 at 9/25/23 10:46 AM:
-------------------------------------------------------------------------
Branches
[trunk|https://github.com/instaclustr/cassandra/tree/CASSANDRA-18773-trunk]
[5.0|https://github.com/instaclustr/cassandra/tree/CASSANDRA-18773-5.0]
[4.1|https://github.com/instaclustr/cassandra/tree/CASSANDRA-18773-4.1]
[4.0|https://github.com/instaclustr/cassandra/tree/CASSANDRA-18773-4.0]
builds
[trunk j17 pre-commit|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3219/workflows/deef3892-0dbe-4f8b-b442-86c195663f54]
[trunk j11 pre-commit|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3230/workflows/6b117ebe-050d-4e79-adbd-c5568418612b]
[5.0 j17 pre-commit|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3220/workflows/1bdb19c9-93fd-4320-ac9d-7f5e574e907e]
[4.1 j11 pre-commit|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3228/workflows/38367be4-c947-4de3-9c0e-fadf65f9c2f1]
[4.0 j11 pre-commit|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3204/workflows/c918e015-273c-4f9a-ad1a-4879f4e2d057]
[4.0 j8 pre-commit|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3204/workflows/68102af9-32c3-403a-8bd0-76f1898feca0]
was (Author: smiklosovic):
Branches
[trunk|https://github.com/instaclustr/cassandra/tree/CASSANDRA-18773-trunk]
[5.0|https://github.com/instaclustr/cassandra/tree/CASSANDRA-18773-5.0]
[4.1|https://github.com/instaclustr/cassandra/tree/CASSANDRA-18773-4.1]
[4.0|https://github.com/instaclustr/cassandra/tree/CASSANDRA-18773-4.0]
PRs
[trunk j17 pre-commit|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3219/workflows/deef3892-0dbe-4f8b-b442-86c195663f54]
[trunk j11 pre-commit|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3230/workflows/6b117ebe-050d-4e79-adbd-c5568418612b]
[5.0 j17 pre-commit|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3220/workflows/1bdb19c9-93fd-4320-ac9d-7f5e574e907e]
[4.1 j11 pre-commit|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3228/workflows/38367be4-c947-4de3-9c0e-fadf65f9c2f1]
[4.0 j11 pre-commit|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3204/workflows/c918e015-273c-4f9a-ad1a-4879f4e2d057]
[4.0 j8 pre-commit|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3204/workflows/68102af9-32c3-403a-8bd0-76f1898feca0]
> Compactions are slow
> --------------------
>
> Key: CASSANDRA-18773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18773
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Compaction
> Reporter: Cameron Zemek
> Assignee: Cameron Zemek
> Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
> Attachments: 18773.patch, compact-poc.patch, flamegraph.png, stress.yaml
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> I have noticed that compactions involving a lot of sstables are very slow (for example major compactions). I have attached a cassandra stress profile that can generate such a dataset under ccm. In my local test I have 2567 sstables at 4Mb each.
> I added code to track wall clock time of various parts of the code. One problematic part is ManyToOne constructor. Tracing through the code for every partition creating a ManyToOne for all the sstable iterators for each partition. In my local test get a measy 60Kb/sec read speed, and bottlenecked on single core CPU (since this code is single threaded) with it spending 85% of the wall clock time in ManyToOne constructor.
> As another datapoint to show its the merge iterator part of the code using the cfstats from [https://github.com/instaclustr/cassandra-sstable-tools/] which reads all the sstables but does no merging gets 26Mb/sec read speed.
> Tracking back from ManyToOne call I see this in UnfilteredPartitionIterators::merge
> {code:java}
> for (int i = 0; i < toMerge.size(); i++)
> {
> if (toMerge.get(i) == null)
> {
> if (null == empty)
> empty = EmptyIterators.unfilteredRow(metadata, partitionKey, isReverseOrder);
> toMerge.set(i, empty);
> }
> }
> {code}
> Not sure what purpose of creating these empty rows are. But on a whim I removed all these empty iterators before passing to ManyToOne and then all the wall clock time shifted to CompactionIterator::hasNext() and read speed increased to 1.5Mb/s.
> So there are further bottlenecks in this code path it seems, but the first is this ManyToOne and having to build it for every partition read.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org