You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jai Bheemsen Rao Dhanwada (Jira)" <ji...@apache.org> on 2025/04/21 17:41:00 UTC
[jira] [Updated] (CASSANDRA-20571) CPU Spikes during the Streaming of data
[ https://issues.apache.org/jira/browse/CASSANDRA-20571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jai Bheemsen Rao Dhanwada updated CASSANDRA-20571:
--------------------------------------------------
Since Version: 4.1.1
> CPU Spikes during the Streaming of data
> ---------------------------------------
>
> Key: CASSANDRA-20571
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20571
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Jai Bheemsen Rao Dhanwada
> Priority: Normal
> Attachments: async_profiler_cpu.html
>
>
> Hello Team,
> We are seeing an issue where there is a huge spike in CPU on the node which is streaming data (adding a new node or replacing a node or running a nodetool rebuild). Essentially anytime when there is a Streaming is involved the CPU spike is very huge. This does not happen in all the clusters but we occasionally see this issue on specific cluster.
>
> C* version: 4.1.6 (> 4.1.0)
> Schema: All the tables use counter data types.
> CPU Cores: 16
>
> The same worksloads + clusters types do not show this behavior with the 4.0.x version of cassandra, hence we suspect something changed in 4.1.6. Looking at the top threads it's mostly the StreamDeserialize + compaction.
> {code:java}
> top - 17:01:29 up 18:42, 2 users, load average: 51.75, 13.61, 4.79
> Threads: 741 total, 54 running, 687 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 91.5 us, 4.9 sy, 0.0 ni, 1.4 id, 0.7 wa, 1.1 hi, 0.4 si, 0.0 st
> MiB Mem : 31176.5 total, 8762.5 free, 11028.0 used, 11386.0 buff/cache
> MiB Swap: 0.0 total, 0.0 free, 0.0 used. 19334.3 avail Mem
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 305763 xxxxxx 20 0 18.6g 9.8g 446524 R 30.8 32.2 0:04.69 Stream-Deserial
> 305815 xxxxxx 20 0 18.6g 9.8g 446524 R 28.6 32.2 0:04.81 Stream-Deserial
> 300600 xxxxxx 20 0 18.6g 9.8g 446524 R 27.9 32.2 0:04.73 CompactionExecu
> 305678 xxxxxx 20 0 18.6g 9.8g 446524 R 27.9 32.2 0:03.98 Stream-Deserial
> 305602 xxxxxx 20 0 18.6g 9.8g 446524 R 27.6 32.2 0:04.65 Stream-Deserial
> 305563 xxxxxx 20 0 18.6g 9.8g 446524 R 27.3 32.2 0:04.02 CompactionExecu
> 305687 xxxxxx 20 0 18.6g 9.8g 446524 R 26.9 32.2 0:04.28 Stream-Deserial
> 305707 xxxxxx 20 0 18.6g 9.8g 446524 S 26.9 32.2 0:04.29 Stream-Deserial
> 305714 xxxxxx 20 0 18.6g 9.8g 446524 R 26.9 32.2 0:04.91 Stream-Deserial
> 305569 xxxxxx 20 0 18.6g 9.8g 446524 R 26.6 32.2 0:05.69 Stream-Deserial
> 305771 xxxxxx 20 0 18.6g 9.8g 446524 R 26.6 32.2 0:03.99 Stream-Deserial
> 305817 xxxxxx 20 0 18.6g 9.8g 446524 R 26.3 32.2 0:03.79 Stream-Deserial
> 305566 xxxxxx 20 0 18.6g 9.8g 446524 R 26.0 32.2 0:04.64 CompactionExecu {code}
> Initial Hypothesis was if streaming_stats are playing a role here based on: https://issues.apache.org/jira/browse/CASSANDRA-18110. However we turned the streaming_stats: false and still see a spike in CPU. Post the streaming is complete the cluster is back to normal state where we don't see a spike in CPU but we would like to understand what's causing the huge CPU spikes. I have profiler attached during the time of CPU.
> Please let me know if you need any other details.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org