[jira] [Commented] (CASSANDRA-19429) Remove lock contention generated by getCapacity function in SSTableReader

["Dmitry Konstantinov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-19429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18004229#comment-18004229 ] 

Dmitry Konstantinov commented on CASSANDRA-19429:
-------------------------------------------------

Hi, sorry for the delay, I am returning to this issue. So, my theory is that issue happens when we have an active eviction from KeyCache due to compaction or due to full capacity usage and sized based eviction. To emulate it I have taken the original test and adjusted it a bit to increase probability of the mentioned use cases:
 * I have reduced key cache capacity to 50MiB. By default it is usually 100MiB but I just want to wait less time for key cache saturation and eviction activated, it is a normal scenario when the cache is full. I use ./bin/nodetool info to check the current key cache size and ensure that it is eventually full.
 * I have altered table definition to change compaction type to Leveled as more intense.
 * As I mentioned - compaction is a one of key cache eviction contributors, so I intentionally have it on.
 * The test is intentionally mixing reads and writes, because writes via compaction can cause locks which impact reads

I have executed initially such test using 5.x (trunk) version. The issue is not observed for it (there is no lock contention in async profiler lock profile) because due to refactoring done in CASSANDRA-17056 starting from 5.0 there is no anymore keyCache.getCapacity() == 0 check which caused the issue.

I am going to check 4.1.9 now.

> Remove lock contention generated by getCapacity function in SSTableReader
> -------------------------------------------------------------------------
>
>                 Key: CASSANDRA-19429
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-19429
>             Project: Apache Cassandra
>          Issue Type: Bug
>          Components: Local/SSTable
>            Reporter: Dipietro Salvatore
>            Assignee: Dipietro Salvatore
>            Priority: Normal
>             Fix For: 4.0.x, 4.1.x
>
>         Attachments: Screenshot 2024-02-26 at 10.27.10.png, Screenshot 2024-02-27 at 11.29.41.png, Screenshot 2024-03-19 at 15.22.50.png, asprof_cass4.1.3__lock_20240216052912lock.html, image-2024-03-08-15-51-30-439.png, image-2024-03-08-15-52-07-902.png, image-2025-04-21-19-00-55-200.png, image-2025-04-21-19-02-41-725.png, wallclock_profile_with_lock_issue.html
>
>          Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Profiling Cassandra 4.1.3 on large AWS instances, a high number of lock acquires is measured in the `getCapacity` function from `org/apache/cassandra/cache/InstrumentingCache` (1.9M lock acquires per 60 seconds). Based on our tests on r8g.24xlarge instances (using Ubuntu 22.04), this limits the CPU utilization of the system to under 50% when testing at full load and therefore limits the achieved throughput.
> Removing the lock contention from the SSTableReader.java file by replacing the call to `getCapacity` with `size` achieves up to 2.95x increase in throughput on r8g.24xlarge and 2x on r7i.24xlarge:
> |Instance type|Cass 4.1.3|Cass 4.1.3 patched|
> |r8g.24xlarge|168k ops|496k ops (2.95x)|
> |r7i.24xlarge|153k ops|304k ops (1.98x)|
>  
> Instructions to reproduce:
> {code:java}
> ## Requirements for Ubuntu 22.04
> sudo apt install -y ant git openjdk-11-jdk
> ## Build and run
> CASSANDRA_USE_JDK11=true ant realclean && CASSANDRA_USE_JDK11=true ant jar && CASSANDRA_USE_JDK11=true ant stress-build  && rm -rf data && bin/cassandra -f -R
> # Run
> bin/cqlsh -e 'drop table if exists keyspace1.standard1;' && \
> bin/cqlsh -e 'drop keyspace if exists keyspace1;' && \
> bin/nodetool clearsnapshot --all && tools/bin/cassandra-stress write n=10000000 cl=ONE -rate threads=384 -node 127.0.0.1 -log file=cload.log -graph file=cload.html && \
> bin/nodetool compact keyspace1   && sleep 30s && \
> tools/bin/cassandra-stress mixed ratio\(write=10,read=90\) duration=10m cl=ONE -rate threads=406 -node localhost -log file=result.log -graph file=graph.html
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org