You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Krishna Chaitanya <bn...@gmail.com> on 2013/12/04 18:05:45 UTC
OutOfMemory Java Heap Space error on startup...
Hello,
I am currently using Cassandra-2.0.0 on OpenSuse for storing
netflow packets that are seen on my ethernet interface. I deliberately
tried to test Cassandra with heavy data and it ran fine for about 30 mins
after which it crashed with OutOfMemory error. I set up a two-node cluster
to which this data is getting stored with replication_factor 1. Now,
Cassandra is not even starting up. The log is given below for your
reference.
Can I solve
this problem by tweaking JVM OPTS? If yes, which all and how? How can I be
sure that it is not someother issue like corrupted commit log headers, etc.
so as to prevent these errors in the future? I am on a 32-bit OpenSuse i5
machine with 4G RAM.
Here is the output when I try to start Cassandra:-
linux-0cpn:~/bnsk/
experimentation/apache-cassandra-2.0.0/bin # ./cassandra -f &
[1] 984
linux-0cpn:~/bnsk/experimentation/apache-cassandra-2.0.0/bin #
./../conf/cassandra-env.sh: line 137: elseError:: command not found
xss = -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn256M
-XX:+HeapDumpOnOutOfMemoryError -Xss256k
INFO 03:46:43,178 Logging initialized
INFO 03:46:43,188 32bit JVM detected. It is recommended to run Cassandra
on a 64bit JVM for better performance.
INFO 03:46:43,189 JVM vendor/version: Java HotSpot(TM) Server VM/1.7.0_45
INFO 03:46:43,189 Heap size: 1046937600/1046937600
INFO 03:46:43,189 Classpath:
./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-2.0.0.jar:./../lib/apache-cassandra-clientutil-2.0.0.jar:./../lib/apache-cassandra-thrift-2.0.0.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/disruptor-3.0.1.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.9.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.11.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar:./../lib/thrift-server-0.3.0.jar:./../lib/jamm-0.2.5.jar
INFO 03:46:43,191 JNA not found. Native methods will be disabled.
INFO 03:46:43,199 Loading settings from
file:/root/bnsk/experimentation/apache-cassandra-2.0.0/conf/cassandra.yaml
INFO 03:46:43,418 Data files directories: [/var/lib/cassandra/data]
INFO 03:46:43,418 Commit log directory: /var/lib/cassandra/commitlog
INFO 03:46:43,418 DiskAccessMode 'auto' determined to be standard,
indexAccessMode is standard
INFO 03:46:43,418 disk_failure_policy is stop
INFO 03:46:43,422 Global memtable threshold is enabled at 332MB
INFO 03:46:43,510 Not using multi-threaded compaction
INFO 03:46:43,660 Initializing key cache with capacity of 49 MBs.
INFO 03:46:43,667 Scheduling key cache save to each 14400 seconds (going
to save all keys).
INFO 03:46:43,668 Initializing row cache with capacity of 0 MBs
INFO 03:46:43,674 Scheduling row cache save to each 0 seconds (going to
save all keys).
INFO 03:46:43,755 Initializing system.schema_triggers
INFO 03:46:43,768 Opening
/var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-104
(57 bytes)
INFO 03:46:43,768 Opening
/var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-105
(57 bytes)
INFO 03:46:43,769 Opening
/var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-103
(57 bytes)
INFO 03:46:43,790 reading saved cache
/var/lib/cassandra/saved_caches/system-schema_triggers-KeyCache-b.db
INFO 03:46:43,798 Initializing system.batchlog
INFO 03:46:43,800 Initializing system.peer_events
INFO 03:46:43,804 Initializing system.compactions_in_progress
INFO 03:46:43,805 Opening
/var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ja-22
(177 bytes)
INFO 03:46:43,805 Opening
/var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ja-23
(42 bytes)
INFO 03:46:43,809 Initializing system.hints
INFO 03:46:43,812 Initializing system.schema_keyspaces
INFO 03:46:43,815 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ja-493
(554 bytes)
INFO 03:46:43,816 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ja-494
(473 bytes)
INFO 03:46:43,818 reading saved cache
/var/lib/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db
INFO 03:46:43,821 Initializing system.range_xfers
INFO 03:46:43,824 Initializing system.schema_columnfamilies
INFO 03:46:43,826 Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-535
(77 bytes)
INFO 03:46:43,827 Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-534
(75 bytes)
INFO 03:46:43,833 Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-533
(8589 bytes)
INFO 03:46:43,841 reading saved cache
/var/lib/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid984.hprof ...
Heap dump file created [15493326 bytes in 0.068 secs]
ERROR 03:46:43,995 Exception encountered during startup
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
at
org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:337)
at
org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:147)
at
org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:267)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:383)
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
at
org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:447)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
at
org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:337)
at
org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:147)
at
org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:267)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:383)
at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
at
org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:447)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
Exception encountered during startup: Java heap space
Thanks in advance...
--
Regards,
BNSK
*.*
Re: OutOfMemory Java Heap Space error on startup...
Posted by Tom van den Berge <to...@drillster.com>.
To start up your node again, you could delete the stored key caches (
/var/lib/cassandra/saved_caches/*).
Regards,
Tom
On Wed, Dec 4, 2013 at 7:46 PM, Krishna Chaitanya <bn...@gmail.com>wrote:
> Hey Nate,
> Thanks for the reply. The link was really good...!!! Looking
> forward to making the necessary changes and trying this approach.
>
> Thanks.
>
> Regards,
> BNSK.
>
>
> On Wed, Dec 4, 2013 at 9:00 AM, Nate McCall <na...@thelastpickle.com>wrote:
>
>> For a limited memory environment, take a look at the following:
>>
>> http://www.opensourceconnections.com/2013/08/31/building-the-perfect-cassandra-test-environment/
>>
>>
>>
>>
>> On Wed, Dec 4, 2013 at 11:05 AM, Krishna Chaitanya <
>> bnsk1990rulz@gmail.com> wrote:
>>
>>> Hello,
>>> I am currently using Cassandra-2.0.0 on OpenSuse for storing
>>> netflow packets that are seen on my ethernet interface. I deliberately
>>> tried to test Cassandra with heavy data and it ran fine for about 30 mins
>>> after which it crashed with OutOfMemory error. I set up a two-node cluster
>>> to which this data is getting stored with replication_factor 1. Now,
>>> Cassandra is not even starting up. The log is given below for your
>>> reference.
>>> Can I solve
>>> this problem by tweaking JVM OPTS? If yes, which all and how? How can I be
>>> sure that it is not someother issue like corrupted commit log headers, etc.
>>> so as to prevent these errors in the future? I am on a 32-bit OpenSuse i5
>>> machine with 4G RAM.
>>>
>>> Here is the output when I try to start Cassandra:-
>>>
>>> linux-0cpn:~/bnsk/
>>> experimentation/apache-cassandra-2.0.0/bin # ./cassandra -f &
>>> [1] 984
>>> linux-0cpn:~/bnsk/experimentation/apache-cassandra-2.0.0/bin #
>>> ./../conf/cassandra-env.sh: line 137: elseError:: command not found
>>> xss = -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
>>> -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn256M
>>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k
>>> INFO 03:46:43,178 Logging initialized
>>> INFO 03:46:43,188 32bit JVM detected. It is recommended to run
>>> Cassandra on a 64bit JVM for better performance.
>>> INFO 03:46:43,189 JVM vendor/version: Java HotSpot(TM) Server
>>> VM/1.7.0_45
>>> INFO 03:46:43,189 Heap size: 1046937600/1046937600
>>> INFO 03:46:43,189 Classpath:
>>> ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-2.0.0.jar:./../lib/apache-cassandra-clientutil-2.0.0.jar:./../lib/apache-cassandra-thrift-2.0.0.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/disruptor-3.0.1.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.9.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.11.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar:./../lib/thrift-server-0.3.0.jar:./../lib/jamm-0.2.5.jar
>>> INFO 03:46:43,191 JNA not found. Native methods will be disabled.
>>> INFO 03:46:43,199 Loading settings from
>>> file:/root/bnsk/experimentation/apache-cassandra-2.0.0/conf/cassandra.yaml
>>> INFO 03:46:43,418 Data files directories: [/var/lib/cassandra/data]
>>> INFO 03:46:43,418 Commit log directory: /var/lib/cassandra/commitlog
>>> INFO 03:46:43,418 DiskAccessMode 'auto' determined to be standard,
>>> indexAccessMode is standard
>>> INFO 03:46:43,418 disk_failure_policy is stop
>>> INFO 03:46:43,422 Global memtable threshold is enabled at 332MB
>>> INFO 03:46:43,510 Not using multi-threaded compaction
>>> INFO 03:46:43,660 Initializing key cache with capacity of 49 MBs.
>>> INFO 03:46:43,667 Scheduling key cache save to each 14400 seconds
>>> (going to save all keys).
>>> INFO 03:46:43,668 Initializing row cache with capacity of 0 MBs
>>> INFO 03:46:43,674 Scheduling row cache save to each 0 seconds (going to
>>> save all keys).
>>> INFO 03:46:43,755 Initializing system.schema_triggers
>>> INFO 03:46:43,768 Opening
>>> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-104
>>> (57 bytes)
>>> INFO 03:46:43,768 Opening
>>> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-105
>>> (57 bytes)
>>> INFO 03:46:43,769 Opening
>>> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-103
>>> (57 bytes)
>>> INFO 03:46:43,790 reading saved cache
>>> /var/lib/cassandra/saved_caches/system-schema_triggers-KeyCache-b.db
>>> INFO 03:46:43,798 Initializing system.batchlog
>>> INFO 03:46:43,800 Initializing system.peer_events
>>> INFO 03:46:43,804 Initializing system.compactions_in_progress
>>> INFO 03:46:43,805 Opening
>>> /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ja-22
>>> (177 bytes)
>>> INFO 03:46:43,805 Opening
>>> /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ja-23
>>> (42 bytes)
>>> INFO 03:46:43,809 Initializing system.hints
>>> INFO 03:46:43,812 Initializing system.schema_keyspaces
>>> INFO 03:46:43,815 Opening
>>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ja-493
>>> (554 bytes)
>>> INFO 03:46:43,816 Opening
>>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ja-494
>>> (473 bytes)
>>> INFO 03:46:43,818 reading saved cache
>>> /var/lib/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db
>>> INFO 03:46:43,821 Initializing system.range_xfers
>>> INFO 03:46:43,824 Initializing system.schema_columnfamilies
>>> INFO 03:46:43,826 Opening
>>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-535
>>> (77 bytes)
>>> INFO 03:46:43,827 Opening
>>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-534
>>> (75 bytes)
>>> INFO 03:46:43,833 Opening
>>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-533
>>> (8589 bytes)
>>> INFO 03:46:43,841 reading saved cache
>>> /var/lib/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db
>>> java.lang.OutOfMemoryError: Java heap space
>>> Dumping heap to java_pid984.hprof ...
>>> Heap dump file created [15493326 bytes in 0.068 secs]
>>> ERROR 03:46:43,995 Exception encountered during startup
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
>>> at
>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
>>> at
>>> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:337)
>>> at
>>> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:147)
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:267)
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411)
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:383)
>>> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
>>> at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
>>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
>>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
>>> at
>>> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:447)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
>>> at
>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
>>> at
>>> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:337)
>>> at
>>> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:147)
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:267)
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411)
>>> at
>>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:383)
>>> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
>>> at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
>>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
>>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
>>> at
>>> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:447)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
>>> Exception encountered during startup: Java heap space
>>>
>>>
>>> Thanks in advance...
>>> --
>>> Regards,
>>> BNSK
>>> *. *
>>>
>>
>>
>>
>> --
>> -----------------
>> Nate McCall
>> Austin, TX
>> @zznate
>>
>> Co-Founder & Sr. Technical Consultant
>> Apache Cassandra Consulting
>> http://www.thelastpickle.com
>>
>
>
>
> --
> Regards,
> BNSK
> *. *
>
--
Drillster BV
Middenburcht 136
3452MT Vleuten
Netherlands
+31 30 755 5330
Open your free account at www.drillster.com
Re: OutOfMemory Java Heap Space error on startup...
Posted by Krishna Chaitanya <bn...@gmail.com>.
Hey Nate,
Thanks for the reply. The link was really good...!!! Looking
forward to making the necessary changes and trying this approach.
Thanks.
Regards,
BNSK.
On Wed, Dec 4, 2013 at 9:00 AM, Nate McCall <na...@thelastpickle.com> wrote:
> For a limited memory environment, take a look at the following:
>
> http://www.opensourceconnections.com/2013/08/31/building-the-perfect-cassandra-test-environment/
>
>
>
>
> On Wed, Dec 4, 2013 at 11:05 AM, Krishna Chaitanya <bnsk1990rulz@gmail.com
> > wrote:
>
>> Hello,
>> I am currently using Cassandra-2.0.0 on OpenSuse for storing
>> netflow packets that are seen on my ethernet interface. I deliberately
>> tried to test Cassandra with heavy data and it ran fine for about 30 mins
>> after which it crashed with OutOfMemory error. I set up a two-node cluster
>> to which this data is getting stored with replication_factor 1. Now,
>> Cassandra is not even starting up. The log is given below for your
>> reference.
>> Can I solve
>> this problem by tweaking JVM OPTS? If yes, which all and how? How can I be
>> sure that it is not someother issue like corrupted commit log headers, etc.
>> so as to prevent these errors in the future? I am on a 32-bit OpenSuse i5
>> machine with 4G RAM.
>>
>> Here is the output when I try to start Cassandra:-
>>
>> linux-0cpn:~/bnsk/
>> experimentation/apache-cassandra-2.0.0/bin # ./cassandra -f &
>> [1] 984
>> linux-0cpn:~/bnsk/experimentation/apache-cassandra-2.0.0/bin #
>> ./../conf/cassandra-env.sh: line 137: elseError:: command not found
>> xss = -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
>> -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn256M
>> -XX:+HeapDumpOnOutOfMemoryError -Xss256k
>> INFO 03:46:43,178 Logging initialized
>> INFO 03:46:43,188 32bit JVM detected. It is recommended to run
>> Cassandra on a 64bit JVM for better performance.
>> INFO 03:46:43,189 JVM vendor/version: Java HotSpot(TM) Server VM/1.7.0_45
>> INFO 03:46:43,189 Heap size: 1046937600/1046937600
>> INFO 03:46:43,189 Classpath:
>> ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-2.0.0.jar:./../lib/apache-cassandra-clientutil-2.0.0.jar:./../lib/apache-cassandra-thrift-2.0.0.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/disruptor-3.0.1.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.9.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.11.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar:./../lib/thrift-server-0.3.0.jar:./../lib/jamm-0.2.5.jar
>> INFO 03:46:43,191 JNA not found. Native methods will be disabled.
>> INFO 03:46:43,199 Loading settings from
>> file:/root/bnsk/experimentation/apache-cassandra-2.0.0/conf/cassandra.yaml
>> INFO 03:46:43,418 Data files directories: [/var/lib/cassandra/data]
>> INFO 03:46:43,418 Commit log directory: /var/lib/cassandra/commitlog
>> INFO 03:46:43,418 DiskAccessMode 'auto' determined to be standard,
>> indexAccessMode is standard
>> INFO 03:46:43,418 disk_failure_policy is stop
>> INFO 03:46:43,422 Global memtable threshold is enabled at 332MB
>> INFO 03:46:43,510 Not using multi-threaded compaction
>> INFO 03:46:43,660 Initializing key cache with capacity of 49 MBs.
>> INFO 03:46:43,667 Scheduling key cache save to each 14400 seconds (going
>> to save all keys).
>> INFO 03:46:43,668 Initializing row cache with capacity of 0 MBs
>> INFO 03:46:43,674 Scheduling row cache save to each 0 seconds (going to
>> save all keys).
>> INFO 03:46:43,755 Initializing system.schema_triggers
>> INFO 03:46:43,768 Opening
>> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-104
>> (57 bytes)
>> INFO 03:46:43,768 Opening
>> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-105
>> (57 bytes)
>> INFO 03:46:43,769 Opening
>> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-103
>> (57 bytes)
>> INFO 03:46:43,790 reading saved cache
>> /var/lib/cassandra/saved_caches/system-schema_triggers-KeyCache-b.db
>> INFO 03:46:43,798 Initializing system.batchlog
>> INFO 03:46:43,800 Initializing system.peer_events
>> INFO 03:46:43,804 Initializing system.compactions_in_progress
>> INFO 03:46:43,805 Opening
>> /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ja-22
>> (177 bytes)
>> INFO 03:46:43,805 Opening
>> /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ja-23
>> (42 bytes)
>> INFO 03:46:43,809 Initializing system.hints
>> INFO 03:46:43,812 Initializing system.schema_keyspaces
>> INFO 03:46:43,815 Opening
>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ja-493
>> (554 bytes)
>> INFO 03:46:43,816 Opening
>> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ja-494
>> (473 bytes)
>> INFO 03:46:43,818 reading saved cache
>> /var/lib/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db
>> INFO 03:46:43,821 Initializing system.range_xfers
>> INFO 03:46:43,824 Initializing system.schema_columnfamilies
>> INFO 03:46:43,826 Opening
>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-535
>> (77 bytes)
>> INFO 03:46:43,827 Opening
>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-534
>> (75 bytes)
>> INFO 03:46:43,833 Opening
>> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-533
>> (8589 bytes)
>> INFO 03:46:43,841 reading saved cache
>> /var/lib/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid984.hprof ...
>> Heap dump file created [15493326 bytes in 0.068 secs]
>> ERROR 03:46:43,995 Exception encountered during startup
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
>> at
>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
>> at
>> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:337)
>> at
>> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:147)
>> at
>> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:267)
>> at
>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411)
>> at
>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:383)
>> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
>> at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
>> at
>> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:447)
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
>> java.lang.OutOfMemoryError: Java heap space
>> at
>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
>> at
>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
>> at
>> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:337)
>> at
>> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:147)
>> at
>> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:267)
>> at
>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411)
>> at
>> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:383)
>> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
>> at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
>> at
>> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:447)
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
>> Exception encountered during startup: Java heap space
>>
>>
>> Thanks in advance...
>> --
>> Regards,
>> BNSK
>> *. *
>>
>
>
>
> --
> -----------------
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
--
Regards,
BNSK
*.*
Re: OutOfMemory Java Heap Space error on startup...
Posted by Nate McCall <na...@thelastpickle.com>.
For a limited memory environment, take a look at the following:
http://www.opensourceconnections.com/2013/08/31/building-the-perfect-cassandra-test-environment/
On Wed, Dec 4, 2013 at 11:05 AM, Krishna Chaitanya
<bn...@gmail.com>wrote:
> Hello,
> I am currently using Cassandra-2.0.0 on OpenSuse for storing
> netflow packets that are seen on my ethernet interface. I deliberately
> tried to test Cassandra with heavy data and it ran fine for about 30 mins
> after which it crashed with OutOfMemory error. I set up a two-node cluster
> to which this data is getting stored with replication_factor 1. Now,
> Cassandra is not even starting up. The log is given below for your
> reference.
> Can I solve
> this problem by tweaking JVM OPTS? If yes, which all and how? How can I be
> sure that it is not someother issue like corrupted commit log headers, etc.
> so as to prevent these errors in the future? I am on a 32-bit OpenSuse i5
> machine with 4G RAM.
>
> Here is the output when I try to start Cassandra:-
>
> linux-0cpn:~/bnsk/
> experimentation/apache-cassandra-2.0.0/bin # ./cassandra -f &
> [1] 984
> linux-0cpn:~/bnsk/experimentation/apache-cassandra-2.0.0/bin #
> ./../conf/cassandra-env.sh: line 137: elseError:: command not found
> xss = -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -Xms1024M -Xmx1024M -Xmn256M
> -XX:+HeapDumpOnOutOfMemoryError -Xss256k
> INFO 03:46:43,178 Logging initialized
> INFO 03:46:43,188 32bit JVM detected. It is recommended to run Cassandra
> on a 64bit JVM for better performance.
> INFO 03:46:43,189 JVM vendor/version: Java HotSpot(TM) Server VM/1.7.0_45
> INFO 03:46:43,189 Heap size: 1046937600/1046937600
> INFO 03:46:43,189 Classpath:
> ./../conf:./../build/classes/main:./../build/classes/thrift:./../lib/antlr-3.2.jar:./../lib/apache-cassandra-2.0.0.jar:./../lib/apache-cassandra-clientutil-2.0.0.jar:./../lib/apache-cassandra-thrift-2.0.0.jar:./../lib/commons-cli-1.1.jar:./../lib/commons-codec-1.2.jar:./../lib/commons-lang-2.6.jar:./../lib/compress-lzf-0.8.4.jar:./../lib/concurrentlinkedhashmap-lru-1.3.jar:./../lib/disruptor-3.0.1.jar:./../lib/guava-13.0.1.jar:./../lib/high-scale-lib-1.1.2.jar:./../lib/jackson-core-asl-1.9.2.jar:./../lib/jackson-mapper-asl-1.9.2.jar:./../lib/jamm-0.2.5.jar:./../lib/jbcrypt-0.3m.jar:./../lib/jline-1.0.jar:./../lib/json-simple-1.1.jar:./../lib/libthrift-0.9.0.jar:./../lib/log4j-1.2.16.jar:./../lib/lz4-1.1.0.jar:./../lib/metrics-core-2.0.3.jar:./../lib/netty-3.5.9.Final.jar:./../lib/servlet-api-2.5-20081211.jar:./../lib/slf4j-api-1.7.2.jar:./../lib/slf4j-log4j12-1.7.2.jar:./../lib/snakeyaml-1.11.jar:./../lib/snappy-java-1.0.5.jar:./../lib/snaptree-0.1.jar:./../lib/thrift-server-0.3.0.jar:./../lib/jamm-0.2.5.jar
> INFO 03:46:43,191 JNA not found. Native methods will be disabled.
> INFO 03:46:43,199 Loading settings from
> file:/root/bnsk/experimentation/apache-cassandra-2.0.0/conf/cassandra.yaml
> INFO 03:46:43,418 Data files directories: [/var/lib/cassandra/data]
> INFO 03:46:43,418 Commit log directory: /var/lib/cassandra/commitlog
> INFO 03:46:43,418 DiskAccessMode 'auto' determined to be standard,
> indexAccessMode is standard
> INFO 03:46:43,418 disk_failure_policy is stop
> INFO 03:46:43,422 Global memtable threshold is enabled at 332MB
> INFO 03:46:43,510 Not using multi-threaded compaction
> INFO 03:46:43,660 Initializing key cache with capacity of 49 MBs.
> INFO 03:46:43,667 Scheduling key cache save to each 14400 seconds (going
> to save all keys).
> INFO 03:46:43,668 Initializing row cache with capacity of 0 MBs
> INFO 03:46:43,674 Scheduling row cache save to each 0 seconds (going to
> save all keys).
> INFO 03:46:43,755 Initializing system.schema_triggers
> INFO 03:46:43,768 Opening
> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-104
> (57 bytes)
> INFO 03:46:43,768 Opening
> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-105
> (57 bytes)
> INFO 03:46:43,769 Opening
> /var/lib/cassandra/data/system/schema_triggers/system-schema_triggers-ja-103
> (57 bytes)
> INFO 03:46:43,790 reading saved cache
> /var/lib/cassandra/saved_caches/system-schema_triggers-KeyCache-b.db
> INFO 03:46:43,798 Initializing system.batchlog
> INFO 03:46:43,800 Initializing system.peer_events
> INFO 03:46:43,804 Initializing system.compactions_in_progress
> INFO 03:46:43,805 Opening
> /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ja-22
> (177 bytes)
> INFO 03:46:43,805 Opening
> /var/lib/cassandra/data/system/compactions_in_progress/system-compactions_in_progress-ja-23
> (42 bytes)
> INFO 03:46:43,809 Initializing system.hints
> INFO 03:46:43,812 Initializing system.schema_keyspaces
> INFO 03:46:43,815 Opening
> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ja-493
> (554 bytes)
> INFO 03:46:43,816 Opening
> /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ja-494
> (473 bytes)
> INFO 03:46:43,818 reading saved cache
> /var/lib/cassandra/saved_caches/system-schema_keyspaces-KeyCache-b.db
> INFO 03:46:43,821 Initializing system.range_xfers
> INFO 03:46:43,824 Initializing system.schema_columnfamilies
> INFO 03:46:43,826 Opening
> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-535
> (77 bytes)
> INFO 03:46:43,827 Opening
> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-534
> (75 bytes)
> INFO 03:46:43,833 Opening
> /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ja-533
> (8589 bytes)
> INFO 03:46:43,841 reading saved cache
> /var/lib/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid984.hprof ...
> Heap dump file created [15493326 bytes in 0.068 secs]
> ERROR 03:46:43,995 Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:337)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:147)
> at
> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:267)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:383)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
> at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:447)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:337)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:147)
> at
> org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:267)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:411)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:383)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
> at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:447)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:227)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
> Exception encountered during startup: Java heap space
>
>
> Thanks in advance...
> --
> Regards,
> BNSK
> *. *
>
--
-----------------
Nate McCall
Austin, TX
@zznate
Co-Founder & Sr. Technical Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com