You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jai Bheemsen Rao Dhanwada (Jira)" <ji...@apache.org> on 2025/04/21 17:41:00 UTC
[jira] [Created] (CASSANDRA-20571) CPU Spikes during the Streaming of data
Jai Bheemsen Rao Dhanwada created CASSANDRA-20571:
-----------------------------------------------------
Summary: CPU Spikes during the Streaming of data
Key: CASSANDRA-20571
URL: https://issues.apache.org/jira/browse/CASSANDRA-20571
Project: Apache Cassandra
Issue Type: Bug
Reporter: Jai Bheemsen Rao Dhanwada
Attachments: async_profiler_cpu.html
Hello Team,
We are seeing an issue where there is a huge spike in CPU on the node which is streaming data (adding a new node or replacing a node or running a nodetool rebuild). Essentially anytime when there is a Streaming is involved the CPU spike is very huge. This does not happen in all the clusters but we occasionally see this issue on specific cluster.
C* version: 4.1.6 (> 4.1.0)
Schema: All the tables use counter data types.
CPU Cores: 16
The same worksloads + clusters types do not show this behavior with the 4.0.x version of cassandra, hence we suspect something changed in 4.1.6. Looking at the top threads it's mostly the StreamDeserialize + compaction.
{code:java}
top - 17:01:29 up 18:42, 2 users, load average: 51.75, 13.61, 4.79
Threads: 741 total, 54 running, 687 sleeping, 0 stopped, 0 zombie
%Cpu(s): 91.5 us, 4.9 sy, 0.0 ni, 1.4 id, 0.7 wa, 1.1 hi, 0.4 si, 0.0 st
MiB Mem : 31176.5 total, 8762.5 free, 11028.0 used, 11386.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 19334.3 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
305763 xxxxxx 20 0 18.6g 9.8g 446524 R 30.8 32.2 0:04.69 Stream-Deserial
305815 xxxxxx 20 0 18.6g 9.8g 446524 R 28.6 32.2 0:04.81 Stream-Deserial
300600 xxxxxx 20 0 18.6g 9.8g 446524 R 27.9 32.2 0:04.73 CompactionExecu
305678 xxxxxx 20 0 18.6g 9.8g 446524 R 27.9 32.2 0:03.98 Stream-Deserial
305602 xxxxxx 20 0 18.6g 9.8g 446524 R 27.6 32.2 0:04.65 Stream-Deserial
305563 xxxxxx 20 0 18.6g 9.8g 446524 R 27.3 32.2 0:04.02 CompactionExecu
305687 xxxxxx 20 0 18.6g 9.8g 446524 R 26.9 32.2 0:04.28 Stream-Deserial
305707 xxxxxx 20 0 18.6g 9.8g 446524 S 26.9 32.2 0:04.29 Stream-Deserial
305714 xxxxxx 20 0 18.6g 9.8g 446524 R 26.9 32.2 0:04.91 Stream-Deserial
305569 xxxxxx 20 0 18.6g 9.8g 446524 R 26.6 32.2 0:05.69 Stream-Deserial
305771 xxxxxx 20 0 18.6g 9.8g 446524 R 26.6 32.2 0:03.99 Stream-Deserial
305817 xxxxxx 20 0 18.6g 9.8g 446524 R 26.3 32.2 0:03.79 Stream-Deserial
305566 xxxxxx 20 0 18.6g 9.8g 446524 R 26.0 32.2 0:04.64 CompactionExecu {code}
Initial Hypothesis was if streaming_stats are playing a role here based on: https://issues.apache.org/jira/browse/CASSANDRA-18110. However we turned the streaming_stats: false and still see a spike in CPU. Post the streaming is complete the cluster is back to normal state where we don't see a spike in CPU but we would like to understand what's causing the huge CPU spikes. I have profiler attached during the time of CPU.
Please let me know if you need any other details.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org