You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Léo FERLIN SUTTON <lf...@mailjet.com.INVALID> on 2018/07/19 14:43:24 UTC
Why and How is Cassandra using all my ram ?
Hello list !
I have a question about cassandra memory usage.
My cassandra nodes are slowly using up all my ram until they get OOM-Killed.
When I check the memory usage with nodetool info the memory
(off-heap+heap) doesn't match what the java process is really using.
I tried to use pmap to see what the jvm was doing with my ram and saw
a total coherent with what nodetool indicates.
The anon-rss for my java/cassandra process is sometimes 10G bigger
than the memory really used by the process. By really used I mean :
what I see with `free`, `ps` (RSS) or even dmesg (anon-rss) after a
node get's OOM-killed.
nodetool info :
```
ID : 24103931-f89c-4d32-bdb8-a86ef0fefb1a
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 1.84 TB
Generation No : 1530695693
Uptime (seconds) : 1313453
Heap Memory (MB) : 8851.83 / 12288.00
Off Heap Memory (MB) : 7356.09
Data Center : europe-west1
Rack : b
Exceptions : 4
Key Cache : entries 8221, size 99.79 MB, capacity 100 MB,
147132654 hits, 236018776 requests, 0.623 recent hit rate, 14400 save
period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0
hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 362366, size 50 MB, capacity 50 MB,
1131153067 hits, 3863836484 requests, 0.293 recent hit rate, 7200 save
period in seconds
Token : (invoke with -T/--tokens to see all 256 tokens)
```
ps output : ```
cassand+ 24546 229 88.8 160765368 27432784 ? SLl Jul04 50236:43
java -Xloggc:/var/log/cassandra/gc.log -XX:+UseG1GC
-XX:G1RSetUpdatingPauseTimePercent=5 -XX:MaxGCPauseMillis=500
-XX:InitiatingHeapOccupancyPercent=70 -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
-XX:+PrintPromotionFailure -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
-XX:+ExitOnOutOfMemoryError -Xms12G -Xmx12G -ea -Xss256k
-XX:+AlwaysPreTouch -XX:-UseBiasedLocking -XX:StringTableSize=1000003
-XX:+UseTLAB -XX:+ResizeTLAB -XX:+PerfDisableSharedMem
-XX:CompileCommandFile=/etc/cassandra/hotspot_compiler
-javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
-XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.rmi.port=7199
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password
-Djava.library.path=/usr/share/cassandra/lib/sigar-bin
-Dlogback.configurationFile=logback.xml
-Dcassandra.logdir=/var/log/cassandra
-Dcassandra.storagedir=/var/lib/cassandra
-Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp
/etc/cassandra:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.4.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jna-4.2.2.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/jstackjunit-0.0.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-jvm-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.44.Final.jar:/usr/share/cassandra/lib/ohc-core-0.4.3.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.3.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.0.jar:/usr/share/cassandra/lib/reporter-config3-3.0.0.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-3.0.16.jar:/usr/share/cassandra/apache-cassandra-thrift-3.0.16.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:
-XX:HeapDumpPath=/var/lib/cassandra/java_1530694780.hprof
-XX:ErrorFile=/var/lib/cassandra/hs_err_1530694780.log
org.apache.cassandra.service.CassandraDaemon
```
oom-kill output :
```
[Thu Jul 19 11:17:37 2018] Out of memory: Kill process 5433 (java)
score 960 or sacrifice child
[Thu Jul 19 11:17:37 2018] Killed process 5433 (java)
total-vm:196334436kB, anon-rss:29361348kB, file-rss:0kB, shmem-rss:0kB
```
pmap | tail -n 1 :
```
total 161041000K
```
We are using cassandra 3.0.16 on Debian stretch with oracle-java :
```
java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)
```
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Why and How is Cassandra using all my ram ?
Posted by Léo FERLIN SUTTON <lf...@mailjet.com.INVALID>.
On Mon, Jul 23, 2018 at 11:44 PM, Mark Rose <ma...@markrose.ca> wrote:
> Hi Léo,
>
> It's possible that glibc is creating too many memory arenas. Are you
> setting/exporting MALLOC_ARENA_MAX to something sane before calling
> the JVM? You can check that in /proc/<pid>/environ.
>
I have checked and the MALLOC_ARENA_MAX is set to `4`.
I will read up the documentation about it before trying ti change it
to make sure I understand the consequences.
> I would also turn on -XX:NativeMemoryTracking=summary and use jcmd to
> check out native memory usage from the JVM's perspective.
>
I will try it on a third of the cluster to see if I noticed anything !
Thank you Mark for your suggestions/
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Why and How is Cassandra using all my ram ?
Posted by Léo FERLIN SUTTON <lf...@mailjet.com.INVALID>.
Hello again,
> It's possible that glibc is creating too many memory arenas. Are you
> setting/exporting MALLOC_ARENA_MAX to something sane before calling
> the JVM? You can check that in /proc/<pid>/environ.
I checked and we have the default value of 4.
> I would also turn on -XX:NativeMemoryTracking=summary and use jcmd to
> check out native memory usage from the JVM's perspective.
>
> -Mark
I have turned on `-XX:NativeMemoryTracking=summary` and have more
information to share.
Right now one of my cassandra nodes is using `22467424` of RSS memory
(around 22GB) according to `ps`.
`nodetool info` tells me it should be using around 17GB :
```
Heap Memory (MB) : 3129.20 / 12288.00
Off Heap Memory (MB) : 5165.15
```
Finally this is the output of the jcmd :
```
sudo -u cassandra jcmd 30952 VM.native_memory summary.diff
30952:
Native Memory Tracking:
Total: reserved=15534476KB +402527KB, committed=14271664KB +418143KB
- Java Heap (reserved=12582912KB, committed=12582912KB)
(mmap: reserved=12582912KB, committed=12582912KB)
- Class (reserved=1120624KB +30810KB,
committed=75960KB +29914KB)
(classes #7034 +231)
(malloc=31088KB +26714KB #166677 +123037)
(mmap: reserved=1089536KB +4096KB,
committed=44872KB +3200KB)
- Thread (reserved=184281KB +37799KB,
committed=184281KB +37799KB)
(thread #627 +144)
(stack: reserved=182728KB +37440KB,
committed=182728KB +37440KB)
(malloc=818KB +190KB #3142 +720)
(arena=735KB +169 #1253 +288)
- Code (reserved=260864KB +3698KB,
committed=67296KB +20210KB)
(malloc=11264KB +3698KB #14716 +4048)
(mmap: reserved=249600KB,
committed=56032KB +16512KB)
- GC (reserved=676609KB +121116KB,
committed=676609KB +121116KB)
(malloc=176897KB +121116KB #163106 +113361)
(mmap: reserved=499712KB, committed=499712KB)
- Compiler (reserved=2017KB +728KB, committed=2017KB +728KB)
(malloc=1887KB +728KB #3621 +1189)
(arena=131KB #3)
- Internal (reserved=654807KB +203186KB,
committed=654803KB +203186KB)
(malloc=654803KB +203186KB #284904 +219680)
(mmap: reserved=4KB, committed=0KB)
- Symbol (reserved=16381KB +140KB, committed=16381KB +140KB)
(malloc=14582KB +140KB #65637 +989)
(arena=1798KB #1)
- Native Memory Tracking (reserved=11060KB +7254KB,
committed=11060KB +7254KB)
(malloc=63KB +14KB #685 +148)
(tracking overhead=10997KB +7240KB)
- Arena Chunk (reserved=312KB -2205KB, committed=312KB -2205KB)
(malloc=312KB -2205KB)
- Unknown (reserved=24608KB, committed=32KB)
(mmap: reserved=24608KB, committed=32KB)
```
I made a baseline shortly after restarting the cassandra node.
Unless I am mistaken somewhere this result is very surprising, the jvm
is using memory that the native memory tracking doesn't see ?
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Why and How is Cassandra using all my ram ?
Posted by Léo FERLIN SUTTON <lf...@mailjet.com.INVALID>.
On Tue, Jul 24, 2018 at 4:04 AM, Dennis Lovely <dl...@aegisco.com> wrote:
> you define the max size of your heap (-Xmx), but you do not define the max
> size of your offheap (MaxMetaspaceSize for jdk 8, PermSize for jdk7), so you
> could occupy all of the memory on the instance.
Yes I think we should set up a MaxMetaspaceSize.
I am still going to try to find out why the ram is being used.
> you should also take into account that the memory size per stack (Xss) is -ontop-
> of what you define for the heap, and offheap, so number of spawned threads
> could be a culprit as well, if you tune your offheap size and keep seeing the same
> trouble. id figure out approximately how many thread stacks are getting created,
> times that by 256k, add that to your heap size, and subtract that number from the total
> amount of memory available to the host to come to a proper offheap size.
I have checked the amount of spawned threads (visible with `nodetool
tpstats` or `ps -T -p <pid>`) and it's too low to be the cause of all
the memory consumption. I have about 650 threads so it's less than 1GB
memory used.
Thank you for the suggestions Dennis.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Why and How is Cassandra using all my ram ?
Posted by Dennis Lovely <dl...@aegisco.com>.
you define the max size of your heap (-Xmx), but you do not define the max
size of your offheap (MaxMetaspaceSize for jdk 8, PermSize for jdk7), so
you could occupy all of the memory on the instance. your system killed the
process to preserve itself. you should also take into account that the
memory size per stack (Xss) is -ontop- of what you define for the heap, and
offheap, so number of spawned threads could be a culprit as well, if you
tune your offheap size and keep seeing the same trouble. id figure out
approximately how many thread stacks are getting created, times that by
256k, add that to your heap size, and subtract that number from the total
amount of memory available to the host to come to a proper offheap size.
On Mon, Jul 23, 2018 at 2:44 PM, Mark Rose <ma...@markrose.ca> wrote:
> On 19 July 2018 at 10:43, Léo FERLIN SUTTON <lf...@mailjet.com.invalid>
> wrote:
> > Hello list !
> >
> > I have a question about cassandra memory usage.
> >
> > My cassandra nodes are slowly using up all my ram until they get
> OOM-Killed.
> >
> > When I check the memory usage with nodetool info the memory
> > (off-heap+heap) doesn't match what the java process is really using.
>
> Hi Léo,
>
> It's possible that glibc is creating too many memory arenas. Are you
> setting/exporting MALLOC_ARENA_MAX to something sane before calling
> the JVM? You can check that in /proc/<pid>/environ.
>
> I would also turn on -XX:NativeMemoryTracking=summary and use jcmd to
> check out native memory usage from the JVM's perspective.
>
> -Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
Re: Why and How is Cassandra using all my ram ?
Posted by Mark Rose <ma...@markrose.ca>.
On 19 July 2018 at 10:43, Léo FERLIN SUTTON <lf...@mailjet.com.invalid> wrote:
> Hello list !
>
> I have a question about cassandra memory usage.
>
> My cassandra nodes are slowly using up all my ram until they get OOM-Killed.
>
> When I check the memory usage with nodetool info the memory
> (off-heap+heap) doesn't match what the java process is really using.
Hi Léo,
It's possible that glibc is creating too many memory arenas. Are you
setting/exporting MALLOC_ARENA_MAX to something sane before calling
the JVM? You can check that in /proc/<pid>/environ.
I would also turn on -XX:NativeMemoryTracking=summary and use jcmd to
check out native memory usage from the JVM's perspective.
-Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org