You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mikhail Mazursky <as...@gmail.com> on 2013/04/16 10:30:46 UTC
C* consumes all RAM
Hello.
C* have been running without any problem for some weeks but now it started
to consume all available ram.
The cluster have very little data in it. There are no errors in logs, CPU
is not loaded at all, jstack shows no deadlocks, there are 83 threads.
Read/write latency is 1-4ms.
The question is: how to find out where is the RAM used?
# nodetool ring keyspace name
Address DC Rack Status State Load
Effective-Ownership Token
114967934986393518082067045023985443411
10.0.1.220 datacenter1 rack1 Up Normal 889.71 KB
100.00% 0
10.0.1.222 datacenter1 rack1 Up Normal 918.92 KB
100.00% 56662868904138943096229641828625627278
10.0.1.221 datacenter1 rack1 Up Normal 1.04 MB
100.00% 114967934986393518082067045023985443411
We use AWS m1.medium - 3.75 RAM, 1 Core
java version "1.7.0_10"
Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)
C* is started with the following command
/usr/java/latest/bin/java -ea
-javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42 -Xms1G -Xmx1G -Xmn512M
-XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC
Thanks in advance.
Mikhail.
Re: C* consumes all RAM
Posted by Mikhail Mazursky <as...@gmail.com>.
Thank you, Aaron.
p.s. we're on 1.1.9 - i forgot to mention that.
2013/4/17 aaron morton <aa...@thelastpickle.com>
> You are probably seeing this http://wiki.apache.org/cassandra/FAQ#mmap
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 16/04/2013, at 8:43 PM, Mikhail Mazursky <as...@gmail.com> wrote:
>
> More details:
>
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> 219 3801 0.7 92.7 6561116 3567016 ? SLl Mar11 372:44
> /usr/java/latest/bin/java
>
> Linux XXX@YYY 3.2.30-49.59.amzn1.x86_64 #1 SMP Wed Oct 3 19:54:33 UTC
> 2012 x86_64 x86_64 x86_64 GNU/Linux
>
> total used free shared buffers cached
> Mem: 3754 3695 59 0 37 75
> -/+ buffers/cache: 3582 171
> Swap: 2055 165 1890
>
>
>
> 2013/4/16 Mikhail Mazursky <as...@gmail.com>
>
>> Hello.
>>
>> C* have been running without any problem for some weeks but now it
>> started to consume all available ram.
>>
>> The cluster have very little data in it. There are no errors in logs, CPU
>> is not loaded at all, jstack shows no deadlocks, there are 83 threads.
>> Read/write latency is 1-4ms.
>>
>> The question is: how to find out where is the RAM used?
>>
>> # nodetool ring keyspace name
>> Address DC Rack Status State Load
>> Effective-Ownership Token
>>
>> 114967934986393518082067045023985443411
>> 10.0.1.220 datacenter1 rack1 Up Normal 889.71 KB
>> 100.00% 0
>> 10.0.1.222 datacenter1 rack1 Up Normal 918.92 KB
>> 100.00% 56662868904138943096229641828625627278
>> 10.0.1.221 datacenter1 rack1 Up Normal 1.04 MB
>> 100.00% 114967934986393518082067045023985443411
>>
>> We use AWS m1.medium - 3.75 RAM, 1 Core
>>
>> java version "1.7.0_10"
>> Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
>> Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)
>>
>> C* is started with the following command
>> /usr/java/latest/bin/java -ea
>> -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
>> -XX:ThreadPriorityPolicy=42 -Xms1G -Xmx1G -Xmn512M
>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC
>>
>> Thanks in advance.
>>
>> Mikhail.
>>
>
>
>
Re: C* consumes all RAM
Posted by aaron morton <aa...@thelastpickle.com>.
You are probably seeing this http://wiki.apache.org/cassandra/FAQ#mmap
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 16/04/2013, at 8:43 PM, Mikhail Mazursky <as...@gmail.com> wrote:
> More details:
>
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> 219 3801 0.7 92.7 6561116 3567016 ? SLl Mar11 372:44 /usr/java/latest/bin/java
>
> Linux XXX@YYY 3.2.30-49.59.amzn1.x86_64 #1 SMP Wed Oct 3 19:54:33 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
>
> total used free shared buffers cached
> Mem: 3754 3695 59 0 37 75
> -/+ buffers/cache: 3582 171
> Swap: 2055 165 1890
>
>
>
> 2013/4/16 Mikhail Mazursky <as...@gmail.com>
> Hello.
>
> C* have been running without any problem for some weeks but now it started to consume all available ram.
>
> The cluster have very little data in it. There are no errors in logs, CPU is not loaded at all, jstack shows no deadlocks, there are 83 threads. Read/write latency is 1-4ms.
>
> The question is: how to find out where is the RAM used?
>
> # nodetool ring keyspace name
> Address DC Rack Status State Load Effective-Ownership Token
> 114967934986393518082067045023985443411
> 10.0.1.220 datacenter1 rack1 Up Normal 889.71 KB 100.00% 0
> 10.0.1.222 datacenter1 rack1 Up Normal 918.92 KB 100.00% 56662868904138943096229641828625627278
> 10.0.1.221 datacenter1 rack1 Up Normal 1.04 MB 100.00% 114967934986393518082067045023985443411
>
> We use AWS m1.medium - 3.75 RAM, 1 Core
>
> java version "1.7.0_10"
> Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
> Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)
>
> C* is started with the following command
> /usr/java/latest/bin/java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1G -Xmx1G -Xmn512M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC
>
> Thanks in advance.
>
> Mikhail.
>
Re: C* consumes all RAM
Posted by Mikhail Mazursky <as...@gmail.com>.
More details:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
219 3801 0.7 92.7 6561116 3567016 ? SLl Mar11 372:44
/usr/java/latest/bin/java
Linux XXX@YYY 3.2.30-49.59.amzn1.x86_64 #1 SMP Wed Oct 3 19:54:33 UTC 2012
x86_64 x86_64 x86_64 GNU/Linux
total used free shared buffers cached
Mem: 3754 3695 59 0 37 75
-/+ buffers/cache: 3582 171
Swap: 2055 165 1890
2013/4/16 Mikhail Mazursky <as...@gmail.com>
> Hello.
>
> C* have been running without any problem for some weeks but now it started
> to consume all available ram.
>
> The cluster have very little data in it. There are no errors in logs, CPU
> is not loaded at all, jstack shows no deadlocks, there are 83 threads.
> Read/write latency is 1-4ms.
>
> The question is: how to find out where is the RAM used?
>
> # nodetool ring keyspace name
> Address DC Rack Status State Load
> Effective-Ownership Token
>
> 114967934986393518082067045023985443411
> 10.0.1.220 datacenter1 rack1 Up Normal 889.71 KB
> 100.00% 0
> 10.0.1.222 datacenter1 rack1 Up Normal 918.92 KB
> 100.00% 56662868904138943096229641828625627278
> 10.0.1.221 datacenter1 rack1 Up Normal 1.04 MB
> 100.00% 114967934986393518082067045023985443411
>
> We use AWS m1.medium - 3.75 RAM, 1 Core
>
> java version "1.7.0_10"
> Java(TM) SE Runtime Environment (build 1.7.0_10-b18)
> Java HotSpot(TM) 64-Bit Server VM (build 23.6-b04, mixed mode)
>
> C* is started with the following command
> /usr/java/latest/bin/java -ea
> -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -Xms1G -Xmx1G -Xmn512M
> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:+UseParNewGC
>
> Thanks in advance.
>
> Mikhail.
>