[jira] [Created] (CASSANDRA-13008) Add vm.max_map_count startup StartupCheck

["Jay Zhuang (JIRA)" <ji...@apache.org>]

Jay Zhuang created CASSANDRA-13008:
--------------------------------------

             Summary: Add vm.max_map_count startup StartupCheck
                 Key: CASSANDRA-13008
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
             Project: Cassandra
          Issue Type: Improvement
          Components: Configuration
            Reporter: Jay Zhuang
            Priority: Minor


It's recommended to set `vm.max_map_count` to 1048575 (CASSANDRA-3563)
When the max_map_count is low, it throws OOM exception, which is hard to link to the real issue of vm.max_map_count.
The problem happened when we tried to remove one node, all the other nodes in cluster crashed. As each node was trying to load more local SSTable files for streaming.

I would suggest to add a StartupCheck for max_map_count, at least it could give a warning message to help the debug.

```
ERROR [MemtableFlushWriter:109] 2016-11-14 00:22:40,598 JVMStabilityInspector.java:117 - JVM state determined to be unstable.  Exiting forcefully due to:
java.lang.OutOfMemoryError: Map failed
    at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ~[na:1.8.0_112]
    at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:377) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:188) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:179) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:344) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$800(BigTableWriter.java:56) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:385) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:169) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:179) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:205) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:412) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:361) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na]
    at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1139) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_112]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_112]
    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
```  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)