You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stefan Miklosovic (Jira)" <ji...@apache.org> on 2024/11/21 10:17:00 UTC
[jira] [Comment Edited] (CASSANDRA-19429) Remove lock contention generated by getCapacity function in SSTableReader
[ https://issues.apache.org/jira/browse/CASSANDRA-19429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17899989#comment-17899989 ]
Stefan Miklosovic edited comment on CASSANDRA-19429 at 11/21/24 10:16 AM:
--------------------------------------------------------------------------
No, I am sorry. Totally swamped these days ... I was thinking about this lately and I think that it would make more sense to focus on trunk only and leave 4.0 / 4.1 be. I understand that we might gain some performance increase in lower branches too but I am not completely sure it is worth it. The type of machines you used for discovering this type of issue are hardly used in common. I do not think clusters with nodes having 96 cores and 768 GB of memory are used often. This is in a realm of hypothetical workloads and even you specifically used this instance type for perf testing we should also take into the consideration the practical benefits. These patches for 4.0 / 4.1 are beyond trivial and we should be cautious about not over-doing that unnecessarily.
What are the performance gains for, let's say, 16 cores with 32 GB memory? Could you measure that?
was (Author: smiklosovic):
No, I am sorry. Totally swamped these days ... I was thinking about this lately and I think that it would make more sense to focus on trunk only and leave 4.0 / 4.1 be. I understand that we might gain some performance increase in lower branches too but I am not completely sure it is worth it. The type of machines you used for discovering this type of issue are hardly used in common. I do not think clusters with nodes having 96 cores and 768 GB of memory are used often. This is in a realm of hypothetical workloads and even you specifically used this instance type for perf testing we should also take into the consideration the practical benefits. These patches for 4.0 / 4.1 are beyond trivial and we should be cautious about not over-doing that unnecessarily.
> Remove lock contention generated by getCapacity function in SSTableReader
> -------------------------------------------------------------------------
>
> Key: CASSANDRA-19429
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19429
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
> Reporter: Dipietro Salvatore
> Assignee: Dipietro Salvatore
> Priority: Normal
> Fix For: 4.0.x, 4.1.x
>
> Attachments: Screenshot 2024-02-26 at 10.27.10.png, Screenshot 2024-02-27 at 11.29.41.png, Screenshot 2024-03-19 at 15.22.50.png, asprof_cass4.1.3__lock_20240216052912lock.html, image-2024-03-08-15-51-30-439.png, image-2024-03-08-15-52-07-902.png
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Profiling Cassandra 4.1.3 on large AWS instances, a high number of lock acquires is measured in the `getCapacity` function from `org/apache/cassandra/cache/InstrumentingCache` (1.9M lock acquires per 60 seconds). Based on our tests on r8g.24xlarge instances (using Ubuntu 22.04), this limits the CPU utilization of the system to under 50% when testing at full load and therefore limits the achieved throughput.
> Removing the lock contention from the SSTableReader.java file by replacing the call to `getCapacity` with `size` achieves up to 2.95x increase in throughput on r8g.24xlarge and 2x on r7i.24xlarge:
> |Instance type|Cass 4.1.3|Cass 4.1.3 patched|
> |r8g.24xlarge|168k ops|496k ops (2.95x)|
> |r7i.24xlarge|153k ops|304k ops (1.98x)|
>
> Instructions to reproduce:
> {code:java}
> ## Requirements for Ubuntu 22.04
> sudo apt install -y ant git openjdk-11-jdk
> ## Build and run
> CASSANDRA_USE_JDK11=true ant realclean && CASSANDRA_USE_JDK11=true ant jar && CASSANDRA_USE_JDK11=true ant stress-build && rm -rf data && bin/cassandra -f -R
> # Run
> bin/cqlsh -e 'drop table if exists keyspace1.standard1;' && \
> bin/cqlsh -e 'drop keyspace if exists keyspace1;' && \
> bin/nodetool clearsnapshot --all && tools/bin/cassandra-stress write n=10000000 cl=ONE -rate threads=384 -node 127.0.0.1 -log file=cload.log -graph file=cload.html && \
> bin/nodetool compact keyspace1 && sleep 30s && \
> tools/bin/cassandra-stress mixed ratio\(write=10,read=90\) duration=10m cl=ONE -rate threads=406 -node localhost -log file=result.log -graph file=graph.html
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org