Memory decline

[Rahul Reddy <ra...@gmail.com>]

Hello,

Im seeing continuous decline in memory on a Cassandra instance used to have
20g free memory 15 days back and now its 15g and continue to go down. Same
instance it caused the cassandra instance crash before. Can you please give
me some pointers to look for which is causing continuous decline in memory

Re: Memory decline

[Rahul Reddy <ra...@gmail.com>]

Hi jeff,
Provided information below.
How can i check how much memory allocated to direct memory for jvm

On Thu, Jun 18, 2020 at 11:38 AM Jeff Jirsa <jj...@gmail.com> wrote:

> Some things that are helpful:
>
> - What version of Cassandra
>

3.11.3

> - How much memory allocated to heap
>
Max_heap_size 8G
Heap_new_size is 800M

> - How much memory allocated to direct memory for the JVM
>

- How much memory on the full system
>
32G

> - Do you have a heap dump?
>
nope, but its enabled in jvm options

>
> - Do you have a heap histogram?
>
nope

> - How much data on disk?
>
185G

> - What are your compression and bloom filter settings (both use off-heap
> memory and scale with size of data on disk, but should not leak)
> Bloom_filter_fp_chance 0.01
>
chunk_length_in_kb 64 and LZ4 compressor

And we have setting file_cache_size_in_mb to 1GB

And we have swap configured to zero

>
> On Thu, Jun 18, 2020 at 7:29 AM Reid Pinchback <rp...@tripadvisor.com>
> wrote:
>
>> Just to confirm, is this memory decline outside of the Cassandra
>> process?  If so, I’d look at crond and at memory held for network traffic.
>> Those are the two areas I’ve seen leak.  If you’ve configured to have
>> swap=0, then you end up in a position where even if the memory usage is
>> stale, nothing can push the stale pages out of the way.
>>
>>
>>
>> *From: *Rahul Reddy <ra...@gmail.com>
>> *Reply-To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
>> *Date: *Thursday, June 18, 2020 at 10:27 AM
>> *To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
>> *Subject: *Memory decline
>>
>>
>>
>> *Message from External Sender*
>>
>> Hello,
>>
>>
>>
>> Im seeing continuous decline in memory on a Cassandra instance used to
>> have 20g free memory 15 days back and now its 15g and continue to go down.
>> Same instance it caused the cassandra instance crash before. Can you please
>> give me some pointers to look for which is causing continuous decline in
>> memory
>>
>

Re: Memory decline

[Jeff Jirsa <jj...@gmail.com>]

Some things that are helpful:

- What version of Cassandra
- How much memory allocated to heap
- How much memory allocated to direct memory for the JVM
- How much memory on the full system
- Do you have a heap dump?
- Do you have a heap histogram?
- How much data on disk?
- What are your compression and bloom filter settings (both use off-heap
memory and scale with size of data on disk, but should not leak)


On Thu, Jun 18, 2020 at 7:29 AM Reid Pinchback <rp...@tripadvisor.com>
wrote:

> Just to confirm, is this memory decline outside of the Cassandra process?
> If so, I’d look at crond and at memory held for network traffic.  Those are
> the two areas I’ve seen leak.  If you’ve configured to have swap=0, then
> you end up in a position where even if the memory usage is stale, nothing
> can push the stale pages out of the way.
>
>
>
> *From: *Rahul Reddy <ra...@gmail.com>
> *Reply-To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Date: *Thursday, June 18, 2020 at 10:27 AM
> *To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Subject: *Memory decline
>
>
>
> *Message from External Sender*
>
> Hello,
>
>
>
> Im seeing continuous decline in memory on a Cassandra instance used to
> have 20g free memory 15 days back and now its 15g and continue to go down.
> Same instance it caused the cassandra instance crash before. Can you please
> give me some pointers to look for which is causing continuous decline in
> memory
>

Re: Memory decline

[Reid Pinchback <rp...@tripadvisor.com>]

Just to confirm, is this memory decline outside of the Cassandra process?  If so, I’d look at crond and at memory held for network traffic.  Those are the two areas I’ve seen leak.  If you’ve configured to have swap=0, then you end up in a position where even if the memory usage is stale, nothing can push the stale pages out of the way.

From: Rahul Reddy <ra...@gmail.com>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Thursday, June 18, 2020 at 10:27 AM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Subject: Memory decline

Message from External Sender
Hello,

Im seeing continuous decline in memory on a Cassandra instance used to have 20g free memory 15 days back and now its 15g and continue to go down. Same instance it caused the cassandra instance crash before. Can you please give me some pointers to look for which is causing continuous decline in memory