You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by vignesh s <vi...@gmail.com> on 2025/02/26 12:27:58 UTC
Cassandra Memory Spikes - Tuning Suggestions?
*Setup:*
I have a Cassandra cluster running in 3 datacenters with 3 nodes each
(total 9 nodes), hosted on GCP.
• *Replication Factor:* 3-3-3
• *Compaction Strategy:* LeveledCompactionStrategy
• *Heap Memory:* 10 GB (Total allocated memory: 32 GB)
• *Off-heap Memory:* around 4 GB
• *Workload:* ~1.5K writes/s per node, ~100 reads/s per node (both using
LOCAL_QUORUM)
*Issue:*
I am observing short-lived memory spikes where total memory usage jumps
from 44% to 85%. These occur periodically and last for a short period.
After monitoring tpstats, I noticed that compaction threads are running
during these spikes.
While I understand that compaction is a fundamental process, these memory
spikes make capacity planning difficult.
I tried adjusting the following settings, but they did not have any effect
on the spikes:
• compaction_throughput_mb_per_sec
• concurrent_compactors
*Questions:*
1. Are there other settings I can tune to reduce memory spikes?
2. Could something else be causing these spikes apart from compaction?
Would appreciate any insights on how to smooth out memory usage.
- vignesh
Re: Cassandra Memory Spikes - Tuning Suggestions?
Posted by manish khandelwal <ma...@gmail.com>.
The section *"Why does top report that Cassandra is using a lot more memory
than the Java heap max?" *on the page
https://cassandra.apache.org/doc/latest/cassandra/overview/faq/index.html
can provide some useful information. I have seen Cassandra tries to take
all the available free RAM using mapped files. For example if I have a 16G
heap and 32G full RAM, memory usage will go till 32GB. And if I have 16G
headp and 64G full RAM, memory usage spikes to 50-56G. This happens mostly
for reads in our case when we are generating reports via spark.
Regards
Manish
On Thu, Feb 27, 2025 at 2:07 PM Dmitry Konstantinov <ne...@gmail.com>
wrote:
> I would recommend checking first for what exact metric do you have drops:
> free memory or available memory. There is a common delusion about free vs
> available memory in Linux: https://www.linuxatemyram.com/
> Overwise if you really have spikes in *used* memory and these are spikes
> in memory used by Cassandra process itself then I would try to
> decompose the memory usage: as Jon mentioned the default and expected
> configuration for Java heap is Xmx=Xms + -XX:+AlwaysPreTouch, so the memory
> for Java heap is allocated in advance and not changing dynamically from OS
> point of view if the settings are in place. In this case there should be
> some other memory region growing, here there are some resources which can
> help to figure out which one:
> https://www.youtube.com/watch?v=c755fFv1Rnk
> https://shipilev.net/jvm/anatomy-quarks/12-native-memory-tracking/
> https://blog.arkey.fr/2020/11/30/off-heap-reconnaissance/
>
> Regards,
> Dmitry
>
>
> On Thu, 27 Feb 2025 at 09:31, vignesh s <vi...@gmail.com> wrote:
>
>> Thanks Bowen and Jon for the clarification and suggestions! I will go
>> through them and dig more.
>>
>> Yes, the JVM heap size is fixed and I can see it is allocated at all
>> times. The spikes I am referring to happen in addition to heap allocated
>> memory.
>> I had tuned heap settings to resolve GC pause issues in the past (Thanks
>> to Jon for the blog on GC tuning!), but the spikes existed even before
>> that. So page cache might be the cause here.
>>
>>
>> -vignesh
>>
>> On Wed, Feb 26, 2025 at 9:01 PM Jon Haddad <jo...@rustyrazorblade.com>
>> wrote:
>>
>>> Can you explain a bit more what you mean by memory spikes?
>>>
>>> The defaults we ship use the same settings for min and max JVM heap
>>> size, so you should see all the memory allocated to the JVM at startup.
>>> Did you change anything here? I don't recommend doing so.
>>>
>>> If you're referring to files in the page cache, that's unavoidable
>>> (today) Compaction reads in data and creates new files, and we do that
>>> through the pread system call, which goes through the page cache. This
>>> could be eliminated if we move compaction to direct io [1], but that's not
>>> something that can be done through config. There's no way to eliminate
>>> files being loaded into the page cache right now, and trying to do so means
>>> fighting your OS and the optimizations it provides. vmstat [2] can be
>>> helpful to understand what's going on here. There's a good
>>> reference, Linux Performance Analysis in 60,000 Milliseconds [3], which
>>> shows how to use it as well as other tools.
>>>
>>> Jon
>>>
>>> [1] https://issues.apache.org/jira/browse/CASSANDRA-19987
>>> [2] https://www.man7.org/linux/man-pages/man8/vmstat.8.html
>>> [3]
>>> https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55
>>>
>>>
>>> On Wed, Feb 26, 2025 at 4:30 AM vignesh s <vi...@gmail.com>
>>> wrote:
>>>
>>>> *Setup:*
>>>> I have a Cassandra cluster running in 3 datacenters with 3 nodes each
>>>> (total 9 nodes), hosted on GCP.
>>>> • *Replication Factor:* 3-3-3
>>>> • *Compaction Strategy:* LeveledCompactionStrategy
>>>> • *Heap Memory:* 10 GB (Total allocated memory: 32 GB)
>>>> • *Off-heap Memory:* around 4 GB
>>>> • *Workload:* ~1.5K writes/s per node, ~100 reads/s per node (both
>>>> using LOCAL_QUORUM)
>>>> *Issue:*
>>>> I am observing short-lived memory spikes where total memory usage jumps
>>>> from 44% to 85%. These occur periodically and last for a short period.
>>>> After monitoring tpstats, I noticed that compaction threads are running
>>>> during these spikes.
>>>> While I understand that compaction is a fundamental process, these
>>>> memory spikes make capacity planning difficult.
>>>> I tried adjusting the following settings, but they did not have any
>>>> effect on the spikes:
>>>> • compaction_throughput_mb_per_sec
>>>> • concurrent_compactors
>>>> *Questions:*
>>>> 1. Are there other settings I can tune to reduce memory spikes?
>>>> 2. Could something else be causing these spikes apart from compaction?
>>>>
>>>> Would appreciate any insights on how to smooth out memory usage.
>>>>
>>>> - vignesh
>>>>
>>>
>
> --
> Dmitry Konstantinov
>
Re: Cassandra Memory Spikes - Tuning Suggestions?
Posted by Dmitry Konstantinov <ne...@gmail.com>.
I would recommend checking first for what exact metric do you have drops:
free memory or available memory. There is a common delusion about free vs
available memory in Linux: https://www.linuxatemyram.com/
Overwise if you really have spikes in *used* memory and these are spikes in
memory used by Cassandra process itself then I would try to decompose the
memory usage: as Jon mentioned the default and expected configuration for
Java heap is Xmx=Xms + -XX:+AlwaysPreTouch, so the memory for Java heap is
allocated in advance and not changing dynamically from OS point of view if
the settings are in place. In this case there should be some other memory
region growing, here there are some resources which can help to figure out
which one:
https://www.youtube.com/watch?v=c755fFv1Rnk
https://shipilev.net/jvm/anatomy-quarks/12-native-memory-tracking/
https://blog.arkey.fr/2020/11/30/off-heap-reconnaissance/
Regards,
Dmitry
On Thu, 27 Feb 2025 at 09:31, vignesh s <vi...@gmail.com> wrote:
> Thanks Bowen and Jon for the clarification and suggestions! I will go
> through them and dig more.
>
> Yes, the JVM heap size is fixed and I can see it is allocated at all
> times. The spikes I am referring to happen in addition to heap allocated
> memory.
> I had tuned heap settings to resolve GC pause issues in the past (Thanks
> to Jon for the blog on GC tuning!), but the spikes existed even before
> that. So page cache might be the cause here.
>
>
> -vignesh
>
> On Wed, Feb 26, 2025 at 9:01 PM Jon Haddad <jo...@rustyrazorblade.com>
> wrote:
>
>> Can you explain a bit more what you mean by memory spikes?
>>
>> The defaults we ship use the same settings for min and max JVM heap size,
>> so you should see all the memory allocated to the JVM at startup. Did you
>> change anything here? I don't recommend doing so.
>>
>> If you're referring to files in the page cache, that's unavoidable
>> (today) Compaction reads in data and creates new files, and we do that
>> through the pread system call, which goes through the page cache. This
>> could be eliminated if we move compaction to direct io [1], but that's not
>> something that can be done through config. There's no way to eliminate
>> files being loaded into the page cache right now, and trying to do so means
>> fighting your OS and the optimizations it provides. vmstat [2] can be
>> helpful to understand what's going on here. There's a good
>> reference, Linux Performance Analysis in 60,000 Milliseconds [3], which
>> shows how to use it as well as other tools.
>>
>> Jon
>>
>> [1] https://issues.apache.org/jira/browse/CASSANDRA-19987
>> [2] https://www.man7.org/linux/man-pages/man8/vmstat.8.html
>> [3]
>> https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55
>>
>>
>> On Wed, Feb 26, 2025 at 4:30 AM vignesh s <vi...@gmail.com>
>> wrote:
>>
>>> *Setup:*
>>> I have a Cassandra cluster running in 3 datacenters with 3 nodes each
>>> (total 9 nodes), hosted on GCP.
>>> • *Replication Factor:* 3-3-3
>>> • *Compaction Strategy:* LeveledCompactionStrategy
>>> • *Heap Memory:* 10 GB (Total allocated memory: 32 GB)
>>> • *Off-heap Memory:* around 4 GB
>>> • *Workload:* ~1.5K writes/s per node, ~100 reads/s per node (both
>>> using LOCAL_QUORUM)
>>> *Issue:*
>>> I am observing short-lived memory spikes where total memory usage jumps
>>> from 44% to 85%. These occur periodically and last for a short period.
>>> After monitoring tpstats, I noticed that compaction threads are running
>>> during these spikes.
>>> While I understand that compaction is a fundamental process, these
>>> memory spikes make capacity planning difficult.
>>> I tried adjusting the following settings, but they did not have any
>>> effect on the spikes:
>>> • compaction_throughput_mb_per_sec
>>> • concurrent_compactors
>>> *Questions:*
>>> 1. Are there other settings I can tune to reduce memory spikes?
>>> 2. Could something else be causing these spikes apart from compaction?
>>>
>>> Would appreciate any insights on how to smooth out memory usage.
>>>
>>> - vignesh
>>>
>>
--
Dmitry Konstantinov
Re: Cassandra Memory Spikes - Tuning Suggestions?
Posted by vignesh s <vi...@gmail.com>.
Thanks Bowen and Jon for the clarification and suggestions! I will go
through them and dig more.
Yes, the JVM heap size is fixed and I can see it is allocated at all times.
The spikes I am referring to happen in addition to heap allocated memory.
I had tuned heap settings to resolve GC pause issues in the past (Thanks to
Jon for the blog on GC tuning!), but the spikes existed even before that.
So page cache might be the cause here.
-vignesh
On Wed, Feb 26, 2025 at 9:01 PM Jon Haddad <jo...@rustyrazorblade.com> wrote:
> Can you explain a bit more what you mean by memory spikes?
>
> The defaults we ship use the same settings for min and max JVM heap size,
> so you should see all the memory allocated to the JVM at startup. Did you
> change anything here? I don't recommend doing so.
>
> If you're referring to files in the page cache, that's unavoidable
> (today) Compaction reads in data and creates new files, and we do that
> through the pread system call, which goes through the page cache. This
> could be eliminated if we move compaction to direct io [1], but that's not
> something that can be done through config. There's no way to eliminate
> files being loaded into the page cache right now, and trying to do so means
> fighting your OS and the optimizations it provides. vmstat [2] can be
> helpful to understand what's going on here. There's a good
> reference, Linux Performance Analysis in 60,000 Milliseconds [3], which
> shows how to use it as well as other tools.
>
> Jon
>
> [1] https://issues.apache.org/jira/browse/CASSANDRA-19987
> [2] https://www.man7.org/linux/man-pages/man8/vmstat.8.html
> [3]
> https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55
>
>
> On Wed, Feb 26, 2025 at 4:30 AM vignesh s <vi...@gmail.com>
> wrote:
>
>> *Setup:*
>> I have a Cassandra cluster running in 3 datacenters with 3 nodes each
>> (total 9 nodes), hosted on GCP.
>> • *Replication Factor:* 3-3-3
>> • *Compaction Strategy:* LeveledCompactionStrategy
>> • *Heap Memory:* 10 GB (Total allocated memory: 32 GB)
>> • *Off-heap Memory:* around 4 GB
>> • *Workload:* ~1.5K writes/s per node, ~100 reads/s per node (both using
>> LOCAL_QUORUM)
>> *Issue:*
>> I am observing short-lived memory spikes where total memory usage jumps
>> from 44% to 85%. These occur periodically and last for a short period.
>> After monitoring tpstats, I noticed that compaction threads are running
>> during these spikes.
>> While I understand that compaction is a fundamental process, these memory
>> spikes make capacity planning difficult.
>> I tried adjusting the following settings, but they did not have any
>> effect on the spikes:
>> • compaction_throughput_mb_per_sec
>> • concurrent_compactors
>> *Questions:*
>> 1. Are there other settings I can tune to reduce memory spikes?
>> 2. Could something else be causing these spikes apart from compaction?
>>
>> Would appreciate any insights on how to smooth out memory usage.
>>
>> - vignesh
>>
>
Re: Cassandra Memory Spikes - Tuning Suggestions?
Posted by Jon Haddad <jo...@rustyrazorblade.com>.
Can you explain a bit more what you mean by memory spikes?
The defaults we ship use the same settings for min and max JVM heap size,
so you should see all the memory allocated to the JVM at startup. Did you
change anything here? I don't recommend doing so.
If you're referring to files in the page cache, that's unavoidable (today)
Compaction reads in data and creates new files, and we do that through the
pread system call, which goes through the page cache. This could be
eliminated if we move compaction to direct io [1], but that's not something
that can be done through config. There's no way to eliminate files being
loaded into the page cache right now, and trying to do so means fighting
your OS and the optimizations it provides. vmstat [2] can be helpful to
understand what's going on here. There's a good reference, Linux
Performance Analysis in 60,000 Milliseconds [3], which shows how to use it
as well as other tools.
Jon
[1] https://issues.apache.org/jira/browse/CASSANDRA-19987
[2] https://www.man7.org/linux/man-pages/man8/vmstat.8.html
[3]
https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55
On Wed, Feb 26, 2025 at 4:30 AM vignesh s <vi...@gmail.com> wrote:
> *Setup:*
> I have a Cassandra cluster running in 3 datacenters with 3 nodes each
> (total 9 nodes), hosted on GCP.
> • *Replication Factor:* 3-3-3
> • *Compaction Strategy:* LeveledCompactionStrategy
> • *Heap Memory:* 10 GB (Total allocated memory: 32 GB)
> • *Off-heap Memory:* around 4 GB
> • *Workload:* ~1.5K writes/s per node, ~100 reads/s per node (both using
> LOCAL_QUORUM)
> *Issue:*
> I am observing short-lived memory spikes where total memory usage jumps
> from 44% to 85%. These occur periodically and last for a short period.
> After monitoring tpstats, I noticed that compaction threads are running
> during these spikes.
> While I understand that compaction is a fundamental process, these memory
> spikes make capacity planning difficult.
> I tried adjusting the following settings, but they did not have any effect
> on the spikes:
> • compaction_throughput_mb_per_sec
> • concurrent_compactors
> *Questions:*
> 1. Are there other settings I can tune to reduce memory spikes?
> 2. Could something else be causing these spikes apart from compaction?
>
> Would appreciate any insights on how to smooth out memory usage.
>
> - vignesh
>
Re: Cassandra Memory Spikes - Tuning Suggestions?
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Hi vignesh,
Correlation does not imply causation. I wouldn't work on the assumption
that the memory usage spikes are caused by compactions to start with.
It's best to prove the causal effect first. There's multiple ways to do
this, I'm just throwing in some ideas:
1. taking a heap dump while it is happening and analyse the heap dump
2. triggering manual compaction and observe it, repeat few times at
different time of day/day of week
3. while it is happening, stop the compaction and see whether the
memory usage immediately drop back to baseline
My preferred way is heap dump, because it will tell you exactly what
caused the memory usage spike, not just confirm or rule out one guess at
a time.
I hope this helps.
Cheers,
Bowen
On 26/02/2025 12:27, vignesh s wrote:
> *Setup:*
> I have a Cassandra cluster running in 3 datacenters with 3 nodes each
> (total 9 nodes), hosted on GCP.
> • *Replication Factor:* 3-3-3
> • *Compaction Strategy:* LeveledCompactionStrategy
> • *Heap Memory:* 10 GB (Total allocated memory: 32 GB)
> • *Off-heap Memory:* around 4 GB
> • *Workload:* ~1.5K writes/s per node, ~100 reads/s per node (both
> using LOCAL_QUORUM)
> *Issue:*
> I am observing short-lived memory spikes where total memory usage
> jumps from 44% to 85%. These occur periodically and last for a short
> period.
> After monitoring tpstats, I noticed that compaction threads are
> running during these spikes.
> While I understand that compaction is a fundamental process, these
> memory spikes make capacity planning difficult.
> I tried adjusting the following settings, but they did not have any
> effect on the spikes:
> • compaction_throughput_mb_per_sec
> • concurrent_compactors
> *Questions:*
> 1. Are there other settings I can tune to reduce memory spikes?
> 2. Could something else be causing these spikes apart from compaction?
>
> Would appreciate any insights on how to smooth out memory usage.
>
> - vignesh