You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by 李建奇 <li...@jd.com> on 2015/01/06 08:25:44 UTC
答复: Cassandra consuming whole RAM (64 G)
Hi ,
There is a “possible memory leak “ issue with c* 2.1.2. https://issues.apache.org/jira/browse/CASSANDRA-8248
It happen with our c* 2.1.2 cluster . In /proc/{pid}/maps there are a lot of deleted file maps .
_____
Stephen li
发件人: Joe Ramsey [mailto:joe.ramsey@mac.com]
发送时间: 2015年1月6日 15:12
收件人: user@cassandra.apache.org
主题: Re: Cassandra consuming whole RAM (64 G)
Thanks Rahul and good luck! I’m really curious to hear what the result is.
On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj <ra...@indiamart.com> wrote:
Thanks for your response.. i will get back to you with my findings.
On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <jo...@mac.com> wrote:
That should be “writing too many bytes” not “waiting too many bytes” just for clarity’s sake.
On Jan 6, 2015, at 2:03 AM, Joe Ramsey <jo...@mac.com> wrote:
I’m not an expert. Really just learning this myself but it looks like according to the stack you’re getting an exception waiting too many bytes to the commit log.
That’s controlled by commit_log_segment_size_in_mb setting. The maximum write size that C* will allow is half of the value set for this parameter so if it’s set for 32 (default) that means your max write would be 16MB. (And that’s what’s getting reported in the stack trace. You’re writing 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing the exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and see if the error goes away. It really should get tuned for the amount of writes but that’ll tell you at least if this is the issue.
Let me know how it goes!
Joe
On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <ra...@indiamart.com> wrote:
Hi Joe,
PFB output of system.log
tail -n 100 system.log
INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555 CompactionTask.java:251 - Compacted 4 sstables to [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,]. 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms = 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge counts were {1:246186, 2:11302, 3:2259, 4:538, }
INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579 CompactionTask.java:251 - Compacted 4 sstables to [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,]. 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total partitions merged to 1. Partition merge counts were {1:1, 2:3, }
WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 433053, exceeding specified threshold of 5120 by 427933.
INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 433819, exceeding specified threshold of 5120 by 428699.
INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 436755, exceeding specified threshold of 5120 by 431635.
INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 - ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_glusr_daily_activity_count] is of size 60021, exceeding specified threshold of 5120 by 54901.
INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 517769, exceeding specified threshold of 5120 by 512649.
INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13220136, exceeding specified threshold of 5120 by 13215016.
WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
... 4 common frames omitted
INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13136950, exceeding specified threshold of 5120 by 13131830.
INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129 ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 (24%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325 - Writing Memtable-im_csl_log@1246880835(90297505 serialized bytes, 1530359 ops, 20%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db (21516225 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db (361277 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db (543861 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db (481902 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db (460053 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db (395084 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13480721, exceeding specified threshold of 5120 by 13475601.
WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
... 4 common frames omitted
INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
Regards:
Rahul Bhardwaj
On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <ra...@indiamart.com> wrote:
Hi Joe,
PFA heap dump
regards:
Rahul Bhardwaj
On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <jo...@mac.com> wrote:
Did you try generating a heap dump so you can look through it to see what’s actually happened?
On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <ra...@indiamart.com> wrote:
Hi,
We are using cassandra 2.1 version in a cluster of three machines each with 64 GB RAM
The processes are killed by kernel, coz they are eating all memory (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) because we have 64 GB RAM.
Please help.
Regards:
Rahul Bhardwaj
Follow <http://www.indiamart.com/> IndiaMART.com for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU> !!!
Follow <http://www.indiamart.com/> IndiaMART.com for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU> !!!
Follow <http://www.indiamart.com/> IndiaMART.com for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU> !!!