You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Rahul Bhardwaj <ra...@indiamart.com> on 2015/01/06 06:58:41 UTC
Cassandra consuming whole RAM (64 G)
Hi,
We are using cassandra 2.1 version in a cluster of three machines each with
64 GB RAM
The processes are killed by kernel, coz they are eating all memory
(oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
because we have 64 GB RAM.
Please help.
Regards:
Rahul Bhardwaj
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
Re: Cassandra consuming whole RAM (64 G)
Posted by Rahul Bhardwaj <ra...@indiamart.com>.
Hi All,
I applied Cassandra patch for issue 8248 to see what it do.
Now I noticed below errors in my system.log :
ERROR [NonPeriodicTasks:1] 2015-01-07 10:55:48,869 CassandraDaemon.java:153
- Exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.free(Memory.java:300)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:645)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(Unknown Source)
~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
Source) ~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) ~[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) [na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [na:1.7.0_71]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
ERROR [NonPeriodicTasks:1] 2015-01-07 10:55:48,870 CassandraDaemon.java:153
- Exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.free(Memory.java:300)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:645)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(Unknown Source)
~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
Source) ~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source) ~[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) [na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [na:1.7.0_71]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
INFO [CompactionExecutor:7] 2015-01-07 10:55:48,870
ColumnFamilyStore.java:840 - Enqueuing flush of compactions_in_progress:
164 (0%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:3] 2015-01-07 10:55:48,871 Memtable.java:325 -
Writing Memtable-compactions_in_progress@1529300710(0 serialized bytes, 1
ops, 0%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:3] 2015-01-07 10:55:48,878 Memtable.java:364 -
Completed flushing
/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1327-Data.db
(42 bytes) for commitlog position ReplayPosition(segmentId=1420608014522,
position=44984450)
INFO [CompactionExecutor:8] 2015-01-07 10:55:48,879
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1325-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1324-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1326-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1327-Data.db')]
ERROR [CompactionExecutor:7] 2015-01-07 10:55:48,880
CassandraDaemon.java:153 - Exception in thread
Thread[CompactionExecutor:7,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.free(Memory.java:300)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.obs.OffHeapBitSet.close(OffHeapBitSet.java:143)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.BloomFilter.close(BloomFilter.java:116)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:345)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableRewriter.abort(SSTableRewriter.java:198)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(Unknown Source)
~[na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) [na:1.7.0_71]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [na:1.7.0_71]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
*Please help in knowing the reason for these errors.*
regards:
Rahul Bhardwaj
On Tue, Jan 6, 2015 at 10:46 PM, Ryan Svihla <rs...@foundev.pro> wrote:
> That even with that patch you'll likely run heap pressure with batches of
> that size, so either increase your heap and take the GC hit on CPU (and
> have longer GCs) or don't use large batches.
>
> The batch conversation is a bigger one which I discuss here
> http://lostechies.com/ryansvihla/2014/08/28/cassandra-batch-loading-without-the-batch-keyword/
>
> On Tue, Jan 6, 2015 at 10:07 AM, Rahul Bhardwaj <
> rahul.bhardwaj@indiamart.com> wrote:
>
>> Thanks Ryan... We will keep ur valuable suggestion in resolving this
>> issue.. But what is your take on Cassandra patch for issue 8248 to resolve
>> this.
>>
>>
>> On Tuesday, January 6, 2015, Ryan Svihla <rs...@foundev.pro> wrote:
>>
>>> Btw side note here, you're using GIANT Batches, and the logs are
>>> indicating such, this will cause a signficant amount of heap pressure.
>>>
>>> The root cause fix is not to use giant batches in the first place.
>>>
>>> On Tue, Jan 6, 2015 at 4:43 AM, Rahul Bhardwaj <
>>> rahul.bhardwaj@indiamart.com> wrote:
>>>
>>>> Hi Joe..
>>>>
>>>> Thanks for your valuable solution.. it worked.
>>>>
>>>> But for this problem
>>>>
>>>> *The processes are killed by kernel, coz they are eating all memory
>>>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
>>>> because we have 64 GB RAM.*
>>>>
>>>> Should I apply patch given for issue
>>>> https://issues.apache.org/jira/browse/CASSANDRA-8248 ??
>>>>
>>>>
>>>> regards:
>>>> Rahul Bhardwaj
>>>>
>>>> On Tue, Jan 6, 2015 at 12:42 PM, Joe Ramsey <jo...@mac.com> wrote:
>>>>
>>>>> Thanks Rahul and good luck! I’m really curious to hear what the
>>>>> result is.
>>>>>
>>>>>
>>>>> On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj <
>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>
>>>>> Thanks for your response.. i will get back to you with my findings.
>>>>>
>>>>> On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <jo...@mac.com>
>>>>> wrote:
>>>>>
>>>>>> That should be “writing too many bytes” not “waiting too many bytes”
>>>>>> just for clarity’s sake.
>>>>>>
>>>>>> On Jan 6, 2015, at 2:03 AM, Joe Ramsey <jo...@mac.com> wrote:
>>>>>>
>>>>>> I’m not an expert. Really just learning this myself but it looks
>>>>>> like according to the stack you’re getting an exception waiting too many
>>>>>> bytes to the commit log.
>>>>>>
>>>>>> That’s controlled by commit_log_segment_size_in_mb setting. The
>>>>>> maximum write size that C* will allow is half of the value set for this
>>>>>> parameter so if it’s set for 32 (default) that means your max write would
>>>>>> be 16MB. (And that’s what’s getting reported in the stack trace. You’re
>>>>>> writing 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing
>>>>>> the exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and
>>>>>> see if the error goes away. It really should get tuned for the amount of
>>>>>> writes but that’ll tell you at least if this is the issue.
>>>>>>
>>>>>> Let me know how it goes!
>>>>>>
>>>>>> Joe
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <
>>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>>
>>>>>> Hi Joe,
>>>>>>
>>>>>> PFB output of system.log
>>>>>>
>>>>>> tail -n 100 system.log
>>>>>> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555
>>>>>> CompactionTask.java:251 - Compacted 4 sstables to
>>>>>> [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,].
>>>>>> 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms =
>>>>>> 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge
>>>>>> counts were {1:246186, 2:11302, 3:2259, 4:538, }
>>>>>> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579
>>>>>> CompactionTask.java:251 - Compacted 4 sstables to
>>>>>> [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,].
>>>>>> 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total
>>>>>> partitions merged to 1. Partition merge counts were {1:1, 2:3, }
>>>>>> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>>>> 433053, exceeding specified threshold of 5120 by 427933.
>>>>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
>>>>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>>>> 433819, exceeding specified threshold of 5120 by 428699.
>>>>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
>>>>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>>>> 436755, exceeding specified threshold of 5120 by 431635.
>>>>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
>>>>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692
>>>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>>>> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too
>>>>>> large for the maxiumum size of 16777216
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>>>>>> ~[na:1.7.0_71]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
>>>>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 -
>>>>>> ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space:
>>>>>> 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
>>>>>> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for
>>>>>> [clickstream.im_csl_glusr_daily_activity_count] is of size 60021, exceeding
>>>>>> specified threshold of 5120 by 54901.
>>>>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
>>>>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>>>> 517769, exceeding specified threshold of 5120 by 512649.
>>>>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
>>>>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>>>>> 13220136, exceeding specified threshold of 5120 by 13215016.
>>>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092
>>>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>>>> java.lang.RuntimeException: java.lang.IllegalArgumentException:
>>>>>> Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>>>>>> ~[na:1.7.0_71]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>>> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887
>>>>>> bytes is too large for the maxiumum size of 16777216
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> ... 4 common frames omitted
>>>>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
>>>>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>>>>> 13136950, exceeding specified threshold of 5120 by 13131830.
>>>>>> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129
>>>>>> ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 (24%)
>>>>>> on-heap, 0 (0%) off-heap
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131
>>>>>> Memtable.java:325 - Writing Memtable-im_csl_log@1246880835(90297505
>>>>>> serialized bytes, 1530359 ops, 20%/0% of on/off-heap limit)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623
>>>>>> Memtable.java:364 - Completed flushing
>>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db
>>>>>> (21516225 bytes) for commitlog position
>>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625
>>>>>> Memtable.java:325 - Writing
>>>>>> Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840
>>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723
>>>>>> Memtable.java:364 - Completed flushing
>>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db
>>>>>> (361277 bytes) for commitlog position
>>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724
>>>>>> Memtable.java:325 - Writing
>>>>>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840
>>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848
>>>>>> Memtable.java:364 - Completed flushing
>>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db
>>>>>> (543861 bytes) for commitlog position
>>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850
>>>>>> Memtable.java:325 - Writing
>>>>>> Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840
>>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988
>>>>>> Memtable.java:364 - Completed flushing
>>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db
>>>>>> (481902 bytes) for commitlog position
>>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989
>>>>>> Memtable.java:325 - Writing
>>>>>> Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840
>>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103
>>>>>> Memtable.java:364 - Completed flushing
>>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db
>>>>>> (460053 bytes) for commitlog position
>>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103
>>>>>> Memtable.java:325 - Writing
>>>>>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840
>>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211
>>>>>> Memtable.java:364 - Completed flushing
>>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db
>>>>>> (395084 bytes) for commitlog position
>>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
>>>>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
>>>>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 -
>>>>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>>>>> 13480721, exceeding specified threshold of 5120 by 13475601.
>>>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537
>>>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>>>> java.lang.RuntimeException: java.lang.IllegalArgumentException:
>>>>>> Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>>>>>> ~[na:1.7.0_71]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>>> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032
>>>>>> bytes is too large for the maxiumum size of 16777216
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> ... 4 common frames omitted
>>>>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
>>>>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853
>>>>>> CompactionTask.java:136 - Compacting
>>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
>>>>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921
>>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,].
>>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s.
>>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532
>>>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>>>> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too
>>>>>> large for the maxiumum size of 16777216
>>>>>> at
>>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>>>>>> ~[na:1.7.0_71]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at
>>>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards:
>>>>>> Rahul Bhardwaj
>>>>>>
>>>>>> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <
>>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>>
>>>>>>> Hi Joe,
>>>>>>>
>>>>>>> PFA heap dump
>>>>>>>
>>>>>>>
>>>>>>> regards:
>>>>>>> Rahul Bhardwaj
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <jo...@mac.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Did you try generating a heap dump so you can look through it to
>>>>>>>> see what’s actually happened?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <
>>>>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> We are using cassandra 2.1 version in a cluster of three machines
>>>>>>>> each with 64 GB RAM
>>>>>>>>
>>>>>>>> The processes are killed by kernel, coz they are eating all memory
>>>>>>>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
>>>>>>>> because we have 64 GB RAM.
>>>>>>>>
>>>>>>>> Please help.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards:
>>>>>>>> Rahul Bhardwaj
>>>>>>>>
>>>>>>>>
>>>>>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest
>>>>>>>> updates on this and more: <https://plus.google.com/+indiamart>
>>>>>>>> <https://www.facebook.com/IndiaMART>
>>>>>>>> <https://twitter.com/IndiaMART> Mobile Channel:
>>>>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>>>>> <http://m.indiamart.com/>
>>>>>>>>
>>>>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>>>>>> kyunki Kaam Yahin Banta Hai
>>>>>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates
>>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>>> Mobile Channel:
>>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>>> <http://m.indiamart.com/>
>>>>>>
>>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>>>> kyunki Kaam Yahin Banta Hai
>>>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates
>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>> Mobile Channel:
>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>> <http://m.indiamart.com/>
>>>>>
>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>>> kyunki Kaam Yahin Banta Hai
>>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>>>> this and more: <https://plus.google.com/+indiamart>
>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>> Mobile Channel:
>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>> <http://m.indiamart.com/>
>>>>
>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>> kyunki Kaam Yahin Banta Hai
>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Thanks,
>>> Ryan Svihla
>>>
>>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>> this and more: <https://plus.google.com/+indiamart>
>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>> Mobile Channel:
>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>> <http://m.indiamart.com/>
>>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
>> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>
>
>
>
> --
>
> Thanks,
> Ryan Svihla
>
>
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
Re: Cassandra consuming whole RAM (64 G)
Posted by Ryan Svihla <rs...@foundev.pro>.
That even with that patch you'll likely run heap pressure with batches of
that size, so either increase your heap and take the GC hit on CPU (and
have longer GCs) or don't use large batches.
The batch conversation is a bigger one which I discuss here
http://lostechies.com/ryansvihla/2014/08/28/cassandra-batch-loading-without-the-batch-keyword/
On Tue, Jan 6, 2015 at 10:07 AM, Rahul Bhardwaj <
rahul.bhardwaj@indiamart.com> wrote:
> Thanks Ryan... We will keep ur valuable suggestion in resolving this
> issue.. But what is your take on Cassandra patch for issue 8248 to resolve
> this.
>
>
> On Tuesday, January 6, 2015, Ryan Svihla <rs...@foundev.pro> wrote:
>
>> Btw side note here, you're using GIANT Batches, and the logs are
>> indicating such, this will cause a signficant amount of heap pressure.
>>
>> The root cause fix is not to use giant batches in the first place.
>>
>> On Tue, Jan 6, 2015 at 4:43 AM, Rahul Bhardwaj <
>> rahul.bhardwaj@indiamart.com> wrote:
>>
>>> Hi Joe..
>>>
>>> Thanks for your valuable solution.. it worked.
>>>
>>> But for this problem
>>>
>>> *The processes are killed by kernel, coz they are eating all memory
>>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
>>> because we have 64 GB RAM.*
>>>
>>> Should I apply patch given for issue
>>> https://issues.apache.org/jira/browse/CASSANDRA-8248 ??
>>>
>>>
>>> regards:
>>> Rahul Bhardwaj
>>>
>>> On Tue, Jan 6, 2015 at 12:42 PM, Joe Ramsey <jo...@mac.com> wrote:
>>>
>>>> Thanks Rahul and good luck! I’m really curious to hear what the result
>>>> is.
>>>>
>>>>
>>>> On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj <
>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>
>>>> Thanks for your response.. i will get back to you with my findings.
>>>>
>>>> On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <jo...@mac.com> wrote:
>>>>
>>>>> That should be “writing too many bytes” not “waiting too many bytes”
>>>>> just for clarity’s sake.
>>>>>
>>>>> On Jan 6, 2015, at 2:03 AM, Joe Ramsey <jo...@mac.com> wrote:
>>>>>
>>>>> I’m not an expert. Really just learning this myself but it looks like
>>>>> according to the stack you’re getting an exception waiting too many bytes
>>>>> to the commit log.
>>>>>
>>>>> That’s controlled by commit_log_segment_size_in_mb setting. The
>>>>> maximum write size that C* will allow is half of the value set for this
>>>>> parameter so if it’s set for 32 (default) that means your max write would
>>>>> be 16MB. (And that’s what’s getting reported in the stack trace. You’re
>>>>> writing 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing
>>>>> the exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and
>>>>> see if the error goes away. It really should get tuned for the amount of
>>>>> writes but that’ll tell you at least if this is the issue.
>>>>>
>>>>> Let me know how it goes!
>>>>>
>>>>> Joe
>>>>>
>>>>>
>>>>>
>>>>> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <
>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>
>>>>> Hi Joe,
>>>>>
>>>>> PFB output of system.log
>>>>>
>>>>> tail -n 100 system.log
>>>>> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555
>>>>> CompactionTask.java:251 - Compacted 4 sstables to
>>>>> [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,].
>>>>> 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms =
>>>>> 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge
>>>>> counts were {1:246186, 2:11302, 3:2259, 4:538, }
>>>>> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579
>>>>> CompactionTask.java:251 - Compacted 4 sstables to
>>>>> [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,].
>>>>> 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total
>>>>> partitions merged to 1. Partition merge counts were {1:1, 2:3, }
>>>>> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 -
>>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>>> 433053, exceeding specified threshold of 5120 by 427933.
>>>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883
>>>>> CompactionTask.java:136 - Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
>>>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952
>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,].
>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s.
>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 -
>>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>>> 433819, exceeding specified threshold of 5120 by 428699.
>>>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982
>>>>> CompactionTask.java:136 - Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
>>>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048
>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,].
>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s.
>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 -
>>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>>> 436755, exceeding specified threshold of 5120 by 431635.
>>>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074
>>>>> CompactionTask.java:136 - Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
>>>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139
>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,].
>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s.
>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692
>>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>>> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too
>>>>> large for the maxiumum size of 16777216
>>>>> at
>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>>> Source) ~[na:1.7.0_71]
>>>>> at
>>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167
>>>>> CompactionTask.java:136 - Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
>>>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241
>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,].
>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 -
>>>>> ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space:
>>>>> 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
>>>>> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 -
>>>>> Batch of prepared statements for
>>>>> [clickstream.im_csl_glusr_daily_activity_count] is of size 60021, exceeding
>>>>> specified threshold of 5120 by 54901.
>>>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274
>>>>> CompactionTask.java:136 - Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
>>>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348
>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,].
>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s.
>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 -
>>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>>> 517769, exceeding specified threshold of 5120 by 512649.
>>>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374
>>>>> CompactionTask.java:136 - Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
>>>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445
>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,].
>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s.
>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 -
>>>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>>>> 13220136, exceeding specified threshold of 5120 by 13215016.
>>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092
>>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>>> java.lang.RuntimeException: java.lang.IllegalArgumentException:
>>>>> Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
>>>>> at
>>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>>> Source) ~[na:1.7.0_71]
>>>>> at
>>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887
>>>>> bytes is too large for the maxiumum size of 16777216
>>>>> at
>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> ... 4 common frames omitted
>>>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472
>>>>> CompactionTask.java:136 - Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
>>>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538
>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,].
>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s.
>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 -
>>>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>>>> 13136950, exceeding specified threshold of 5120 by 13131830.
>>>>> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129
>>>>> ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 (24%)
>>>>> on-heap, 0 (0%) off-heap
>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131
>>>>> Memtable.java:325 - Writing Memtable-im_csl_log@1246880835(90297505
>>>>> serialized bytes, 1530359 ops, 20%/0% of on/off-heap limit)
>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623
>>>>> Memtable.java:364 - Completed flushing
>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db
>>>>> (21516225 bytes) for commitlog position
>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625
>>>>> Memtable.java:325 - Writing
>>>>> Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840 serialized
>>>>> bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723
>>>>> Memtable.java:364 - Completed flushing
>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db
>>>>> (361277 bytes) for commitlog position
>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724
>>>>> Memtable.java:325 - Writing
>>>>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840
>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848
>>>>> Memtable.java:364 - Completed flushing
>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db
>>>>> (543861 bytes) for commitlog position
>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850
>>>>> Memtable.java:325 - Writing
>>>>> Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840
>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988
>>>>> Memtable.java:364 - Completed flushing
>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db
>>>>> (481902 bytes) for commitlog position
>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989
>>>>> Memtable.java:325 - Writing
>>>>> Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840
>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103
>>>>> Memtable.java:364 - Completed flushing
>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db
>>>>> (460053 bytes) for commitlog position
>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103
>>>>> Memtable.java:325 - Writing
>>>>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840
>>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211
>>>>> Memtable.java:364 - Completed flushing
>>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db
>>>>> (395084 bytes) for commitlog position
>>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563
>>>>> CompactionTask.java:136 - Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
>>>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633
>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,].
>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s.
>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659
>>>>> CompactionTask.java:136 - Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
>>>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733
>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,].
>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 -
>>>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>>>> 13480721, exceeding specified threshold of 5120 by 13475601.
>>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537
>>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>>> java.lang.RuntimeException: java.lang.IllegalArgumentException:
>>>>> Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
>>>>> at
>>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>>> Source) ~[na:1.7.0_71]
>>>>> at
>>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032
>>>>> bytes is too large for the maxiumum size of 16777216
>>>>> at
>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> ... 4 common frames omitted
>>>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762
>>>>> CompactionTask.java:136 - Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
>>>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825
>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,].
>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s.
>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853
>>>>> CompactionTask.java:136 - Compacting
>>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
>>>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921
>>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,].
>>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s.
>>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532
>>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>>> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too
>>>>> large for the maxiumum size of 16777216
>>>>> at
>>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>>> Source) ~[na:1.7.0_71]
>>>>> at
>>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at
>>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>>
>>>>>
>>>>>
>>>>> Regards:
>>>>> Rahul Bhardwaj
>>>>>
>>>>> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <
>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>
>>>>>> Hi Joe,
>>>>>>
>>>>>> PFA heap dump
>>>>>>
>>>>>>
>>>>>> regards:
>>>>>> Rahul Bhardwaj
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <jo...@mac.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Did you try generating a heap dump so you can look through it to see
>>>>>>> what’s actually happened?
>>>>>>>
>>>>>>>
>>>>>>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <
>>>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We are using cassandra 2.1 version in a cluster of three machines
>>>>>>> each with 64 GB RAM
>>>>>>>
>>>>>>> The processes are killed by kernel, coz they are eating all memory
>>>>>>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
>>>>>>> because we have 64 GB RAM.
>>>>>>>
>>>>>>> Please help.
>>>>>>>
>>>>>>>
>>>>>>> Regards:
>>>>>>> Rahul Bhardwaj
>>>>>>>
>>>>>>>
>>>>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates
>>>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>>>> Mobile Channel:
>>>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>>>> <http://m.indiamart.com/>
>>>>>>>
>>>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>>>>> kyunki Kaam Yahin Banta Hai
>>>>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates
>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>> Mobile Channel:
>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>> <http://m.indiamart.com/>
>>>>>
>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>>> kyunki Kaam Yahin Banta Hai
>>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on
>>>> this and more: <https://plus.google.com/+indiamart>
>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>> Mobile Channel:
>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>> <http://m.indiamart.com/>
>>>>
>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>> kyunki Kaam Yahin Banta Hai
>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>
>>>>
>>>>
>>>
>>>
>>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>>> this and more: <https://plus.google.com/+indiamart>
>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>> Mobile Channel:
>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>> <http://m.indiamart.com/>
>>>
>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
>>> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>
>>
>>
>>
>> --
>>
>> Thanks,
>> Ryan Svihla
>>
>>
>
> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
> this and more: <https://plus.google.com/+indiamart>
> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
> Mobile Channel:
> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
> <https://play.google.com/store/apps/details?id=com.indiamart.m>
> <http://m.indiamart.com/>
>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>
--
Thanks,
Ryan Svihla
Re: Cassandra consuming whole RAM (64 G)
Posted by Rahul Bhardwaj <ra...@indiamart.com>.
Thanks Ryan... We will keep ur valuable suggestion in resolving this
issue.. But what is your take on Cassandra patch for issue 8248 to resolve
this.
On Tuesday, January 6, 2015, Ryan Svihla <rs...@foundev.pro> wrote:
> Btw side note here, you're using GIANT Batches, and the logs are
> indicating such, this will cause a signficant amount of heap pressure.
>
> The root cause fix is not to use giant batches in the first place.
>
> On Tue, Jan 6, 2015 at 4:43 AM, Rahul Bhardwaj <
> rahul.bhardwaj@indiamart.com
> <javascript:_e(%7B%7D,'cvml','rahul.bhardwaj@indiamart.com');>> wrote:
>
>> Hi Joe..
>>
>> Thanks for your valuable solution.. it worked.
>>
>> But for this problem
>>
>> *The processes are killed by kernel, coz they are eating all memory
>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
>> because we have 64 GB RAM.*
>>
>> Should I apply patch given for issue
>> https://issues.apache.org/jira/browse/CASSANDRA-8248 ??
>>
>>
>> regards:
>> Rahul Bhardwaj
>>
>> On Tue, Jan 6, 2015 at 12:42 PM, Joe Ramsey <joe.ramsey@mac.com
>> <javascript:_e(%7B%7D,'cvml','joe.ramsey@mac.com');>> wrote:
>>
>>> Thanks Rahul and good luck! I’m really curious to hear what the result
>>> is.
>>>
>>>
>>> On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com
>>> <javascript:_e(%7B%7D,'cvml','rahul.bhardwaj@indiamart.com');>> wrote:
>>>
>>> Thanks for your response.. i will get back to you with my findings.
>>>
>>> On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <joe.ramsey@mac.com
>>> <javascript:_e(%7B%7D,'cvml','joe.ramsey@mac.com');>> wrote:
>>>
>>>> That should be “writing too many bytes” not “waiting too many bytes”
>>>> just for clarity’s sake.
>>>>
>>>> On Jan 6, 2015, at 2:03 AM, Joe Ramsey <joe.ramsey@mac.com
>>>> <javascript:_e(%7B%7D,'cvml','joe.ramsey@mac.com');>> wrote:
>>>>
>>>> I’m not an expert. Really just learning this myself but it looks like
>>>> according to the stack you’re getting an exception waiting too many bytes
>>>> to the commit log.
>>>>
>>>> That’s controlled by commit_log_segment_size_in_mb setting. The
>>>> maximum write size that C* will allow is half of the value set for this
>>>> parameter so if it’s set for 32 (default) that means your max write would
>>>> be 16MB. (And that’s what’s getting reported in the stack trace. You’re
>>>> writing 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing
>>>> the exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and
>>>> see if the error goes away. It really should get tuned for the amount of
>>>> writes but that’ll tell you at least if this is the issue.
>>>>
>>>> Let me know how it goes!
>>>>
>>>> Joe
>>>>
>>>>
>>>>
>>>> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <
>>>> rahul.bhardwaj@indiamart.com
>>>> <javascript:_e(%7B%7D,'cvml','rahul.bhardwaj@indiamart.com');>> wrote:
>>>>
>>>> Hi Joe,
>>>>
>>>> PFB output of system.log
>>>>
>>>> tail -n 100 system.log
>>>> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555
>>>> CompactionTask.java:251 - Compacted 4 sstables to
>>>> [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,].
>>>> 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms =
>>>> 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge
>>>> counts were {1:246186, 2:11302, 3:2259, 4:538, }
>>>> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579
>>>> CompactionTask.java:251 - Compacted 4 sstables to
>>>> [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,].
>>>> 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total
>>>> partitions merged to 1. Partition merge counts were {1:1, 2:3, }
>>>> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 -
>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>> 433053, exceeding specified threshold of 5120 by 427933.
>>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883
>>>> CompactionTask.java:136 - Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
>>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952
>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,].
>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s.
>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 -
>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>> 433819, exceeding specified threshold of 5120 by 428699.
>>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982
>>>> CompactionTask.java:136 - Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
>>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048
>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,].
>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s.
>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 -
>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>> 436755, exceeding specified threshold of 5120 by 431635.
>>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074
>>>> CompactionTask.java:136 - Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
>>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139
>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,].
>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s.
>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692
>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too
>>>> large for the maxiumum size of 16777216
>>>> at
>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>> Source) ~[na:1.7.0_71]
>>>> at
>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167
>>>> CompactionTask.java:136 - Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
>>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241
>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,].
>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 -
>>>> ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space:
>>>> 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
>>>> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 -
>>>> Batch of prepared statements for
>>>> [clickstream.im_csl_glusr_daily_activity_count] is of size 60021, exceeding
>>>> specified threshold of 5120 by 54901.
>>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274
>>>> CompactionTask.java:136 - Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
>>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348
>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,].
>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s.
>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 -
>>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>>> 517769, exceeding specified threshold of 5120 by 512649.
>>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374
>>>> CompactionTask.java:136 - Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
>>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445
>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,].
>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s.
>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 -
>>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>>> 13220136, exceeding specified threshold of 5120 by 13215016.
>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092
>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>> java.lang.RuntimeException: java.lang.IllegalArgumentException:
>>>> Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
>>>> at
>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>> Source) ~[na:1.7.0_71]
>>>> at
>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887
>>>> bytes is too large for the maxiumum size of 16777216
>>>> at
>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> ... 4 common frames omitted
>>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472
>>>> CompactionTask.java:136 - Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
>>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538
>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,].
>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s.
>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 -
>>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>>> 13136950, exceeding specified threshold of 5120 by 13131830.
>>>> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129
>>>> ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 (24%)
>>>> on-heap, 0 (0%) off-heap
>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131
>>>> Memtable.java:325 - Writing Memtable-im_csl_log@1246880835(90297505
>>>> serialized bytes, 1530359 ops, 20%/0% of on/off-heap limit)
>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623
>>>> Memtable.java:364 - Completed flushing
>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db
>>>> (21516225 bytes) for commitlog position
>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625
>>>> Memtable.java:325 - Writing
>>>> Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840 serialized
>>>> bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723
>>>> Memtable.java:364 - Completed flushing
>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db
>>>> (361277 bytes) for commitlog position
>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724
>>>> Memtable.java:325 - Writing
>>>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840
>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848
>>>> Memtable.java:364 - Completed flushing
>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db
>>>> (543861 bytes) for commitlog position
>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850
>>>> Memtable.java:325 - Writing
>>>> Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840
>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988
>>>> Memtable.java:364 - Completed flushing
>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db
>>>> (481902 bytes) for commitlog position
>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989
>>>> Memtable.java:325 - Writing
>>>> Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840
>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103
>>>> Memtable.java:364 - Completed flushing
>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db
>>>> (460053 bytes) for commitlog position
>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103
>>>> Memtable.java:325 - Writing
>>>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840
>>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211
>>>> Memtable.java:364 - Completed flushing
>>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db
>>>> (395084 bytes) for commitlog position
>>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563
>>>> CompactionTask.java:136 - Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
>>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633
>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,].
>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s.
>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659
>>>> CompactionTask.java:136 - Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
>>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733
>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,].
>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 -
>>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>>> 13480721, exceeding specified threshold of 5120 by 13475601.
>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537
>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>> java.lang.RuntimeException: java.lang.IllegalArgumentException:
>>>> Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
>>>> at
>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>> Source) ~[na:1.7.0_71]
>>>> at
>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032
>>>> bytes is too large for the maxiumum size of 16777216
>>>> at
>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> ... 4 common frames omitted
>>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762
>>>> CompactionTask.java:136 - Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
>>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825
>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,].
>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s.
>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853
>>>> CompactionTask.java:136 - Compacting
>>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
>>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921
>>>> CompactionTask.java:251 - Compacted 1 sstables to
>>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,].
>>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s.
>>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>>> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532
>>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>>> Thread[SharedPool-Worker-1,5,main]: {}
>>>> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too
>>>> large for the maxiumum size of 16777216
>>>> at
>>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>> Source) ~[na:1.7.0_71]
>>>> at
>>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at
>>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>>
>>>>
>>>>
>>>> Regards:
>>>> Rahul Bhardwaj
>>>>
>>>> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <
>>>> rahul.bhardwaj@indiamart.com
>>>> <javascript:_e(%7B%7D,'cvml','rahul.bhardwaj@indiamart.com');>> wrote:
>>>>
>>>>> Hi Joe,
>>>>>
>>>>> PFA heap dump
>>>>>
>>>>>
>>>>> regards:
>>>>> Rahul Bhardwaj
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <joe.ramsey@mac.com
>>>>> <javascript:_e(%7B%7D,'cvml','joe.ramsey@mac.com');>> wrote:
>>>>>
>>>>>> Did you try generating a heap dump so you can look through it to see
>>>>>> what’s actually happened?
>>>>>>
>>>>>>
>>>>>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <
>>>>>> rahul.bhardwaj@indiamart.com
>>>>>> <javascript:_e(%7B%7D,'cvml','rahul.bhardwaj@indiamart.com');>>
>>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We are using cassandra 2.1 version in a cluster of three machines
>>>>>> each with 64 GB RAM
>>>>>>
>>>>>> The processes are killed by kernel, coz they are eating all memory
>>>>>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
>>>>>> because we have 64 GB RAM.
>>>>>>
>>>>>> Please help.
>>>>>>
>>>>>>
>>>>>> Regards:
>>>>>> Rahul Bhardwaj
>>>>>>
>>>>>>
>>>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates
>>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>>> Mobile Channel:
>>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>>> <http://m.indiamart.com/>
>>>>>>
>>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>>>> kyunki Kaam Yahin Banta Hai
>>>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on
>>>> this and more: <https://plus.google.com/+indiamart>
>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>> Mobile Channel:
>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>> <http://m.indiamart.com/>
>>>>
>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>> kyunki Kaam Yahin Banta Hai
>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on
>>> this and more: <https://plus.google.com/+indiamart>
>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>> Mobile Channel:
>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>> <http://m.indiamart.com/>
>>>
>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
>>> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>
>>>
>>>
>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
>> this and more: <https://plus.google.com/+indiamart>
>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>> Mobile Channel:
>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>> <http://m.indiamart.com/>
>>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
>> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>
>
>
>
> --
>
> Thanks,
> Ryan Svihla
>
>
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
Re: Cassandra consuming whole RAM (64 G)
Posted by Ryan Svihla <rs...@foundev.pro>.
Btw side note here, you're using GIANT Batches, and the logs are indicating
such, this will cause a signficant amount of heap pressure.
The root cause fix is not to use giant batches in the first place.
On Tue, Jan 6, 2015 at 4:43 AM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com
> wrote:
> Hi Joe..
>
> Thanks for your valuable solution.. it worked.
>
> But for this problem
>
> *The processes are killed by kernel, coz they are eating all memory
> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
> because we have 64 GB RAM.*
>
> Should I apply patch given for issue
> https://issues.apache.org/jira/browse/CASSANDRA-8248 ??
>
>
> regards:
> Rahul Bhardwaj
>
> On Tue, Jan 6, 2015 at 12:42 PM, Joe Ramsey <jo...@mac.com> wrote:
>
>> Thanks Rahul and good luck! I’m really curious to hear what the result
>> is.
>>
>>
>> On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj <ra...@indiamart.com>
>> wrote:
>>
>> Thanks for your response.. i will get back to you with my findings.
>>
>> On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <jo...@mac.com> wrote:
>>
>>> That should be “writing too many bytes” not “waiting too many bytes”
>>> just for clarity’s sake.
>>>
>>> On Jan 6, 2015, at 2:03 AM, Joe Ramsey <jo...@mac.com> wrote:
>>>
>>> I’m not an expert. Really just learning this myself but it looks like
>>> according to the stack you’re getting an exception waiting too many bytes
>>> to the commit log.
>>>
>>> That’s controlled by commit_log_segment_size_in_mb setting. The maximum
>>> write size that C* will allow is half of the value set for this parameter
>>> so if it’s set for 32 (default) that means your max write would be 16MB.
>>> (And that’s what’s getting reported in the stack trace. You’re writing
>>> 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing the
>>> exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and see
>>> if the error goes away. It really should get tuned for the amount of
>>> writes but that’ll tell you at least if this is the issue.
>>>
>>> Let me know how it goes!
>>>
>>> Joe
>>>
>>>
>>>
>>> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <ra...@indiamart.com>
>>> wrote:
>>>
>>> Hi Joe,
>>>
>>> PFB output of system.log
>>>
>>> tail -n 100 system.log
>>> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555
>>> CompactionTask.java:251 - Compacted 4 sstables to
>>> [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,].
>>> 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms =
>>> 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge
>>> counts were {1:246186, 2:11302, 3:2259, 4:538, }
>>> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579
>>> CompactionTask.java:251 - Compacted 4 sstables to
>>> [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,].
>>> 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total
>>> partitions merged to 1. Partition merge counts were {1:1, 2:3, }
>>> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 -
>>> Batch of prepared statements for [clickstream.business_feed] is of size
>>> 433053, exceeding specified threshold of 5120 by 427933.
>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883
>>> CompactionTask.java:136 - Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952
>>> CompactionTask.java:251 - Compacted 1 sstables to
>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,].
>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s.
>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch
>>> of prepared statements for [clickstream.business_feed] is of size 433819,
>>> exceeding specified threshold of 5120 by 428699.
>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982
>>> CompactionTask.java:136 - Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048
>>> CompactionTask.java:251 - Compacted 1 sstables to
>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,].
>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s.
>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch
>>> of prepared statements for [clickstream.business_feed] is of size 436755,
>>> exceeding specified threshold of 5120 by 431635.
>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074
>>> CompactionTask.java:136 - Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139
>>> CompactionTask.java:251 - Compacted 1 sstables to
>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,].
>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s.
>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692
>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>> Thread[SharedPool-Worker-1,5,main]: {}
>>> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too
>>> large for the maxiumum size of 16777216
>>> at
>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>> Source) ~[na:1.7.0_71]
>>> at
>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167
>>> CompactionTask.java:136 - Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241
>>> CompactionTask.java:251 - Compacted 1 sstables to
>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,].
>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 -
>>> ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space:
>>> 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
>>> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch
>>> of prepared statements for [clickstream.im_csl_glusr_daily_activity_count]
>>> is of size 60021, exceeding specified threshold of 5120 by 54901.
>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274
>>> CompactionTask.java:136 - Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348
>>> CompactionTask.java:251 - Compacted 1 sstables to
>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,].
>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s.
>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch
>>> of prepared statements for [clickstream.business_feed] is of size 517769,
>>> exceeding specified threshold of 5120 by 512649.
>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374
>>> CompactionTask.java:136 - Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445
>>> CompactionTask.java:251 - Compacted 1 sstables to
>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,].
>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s.
>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 -
>>> Batch of prepared statements for [clickstream.im_csl_log] is of size
>>> 13220136, exceeding specified threshold of 5120 by 13215016.
>>> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092
>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>> Thread[SharedPool-Worker-1,5,main]: {}
>>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation
>>> of 16800887 bytes is too large for the maxiumum size of 16777216
>>> at
>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>> Source) ~[na:1.7.0_71]
>>> at
>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887
>>> bytes is too large for the maxiumum size of 16777216
>>> at
>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> ... 4 common frames omitted
>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472
>>> CompactionTask.java:136 - Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538
>>> CompactionTask.java:251 - Compacted 1 sstables to
>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,].
>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s.
>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch
>>> of prepared statements for [clickstream.im_csl_log] is of size 13136950,
>>> exceeding specified threshold of 5120 by 13131830.
>>> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129
>>> ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 (24%)
>>> on-heap, 0 (0%) off-heap
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325
>>> - Writing Memtable-im_csl_log@1246880835(90297505 serialized bytes,
>>> 1530359 ops, 20%/0% of on/off-heap limit)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364
>>> - Completed flushing
>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db
>>> (21516225 bytes) for commitlog position
>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325
>>> - Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840
>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364
>>> - Completed flushing
>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db
>>> (361277 bytes) for commitlog position
>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325
>>> - Writing
>>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840
>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364
>>> - Completed flushing
>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db
>>> (543861 bytes) for commitlog position
>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325
>>> - Writing
>>> Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840
>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364
>>> - Completed flushing
>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db
>>> (481902 bytes) for commitlog position
>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325
>>> - Writing Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840
>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364
>>> - Completed flushing
>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db
>>> (460053 bytes) for commitlog position
>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325
>>> - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840
>>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364
>>> - Completed flushing
>>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db
>>> (395084 bytes) for commitlog position
>>> ReplayPosition(segmentId=1420516895898, position=16708232)
>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563
>>> CompactionTask.java:136 - Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633
>>> CompactionTask.java:251 - Compacted 1 sstables to
>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,].
>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s.
>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659
>>> CompactionTask.java:136 - Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733
>>> CompactionTask.java:251 - Compacted 1 sstables to
>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,].
>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch
>>> of prepared statements for [clickstream.im_csl_log] is of size 13480721,
>>> exceeding specified threshold of 5120 by 13475601.
>>> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537
>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>> Thread[SharedPool-Worker-1,5,main]: {}
>>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation
>>> of 17138032 bytes is too large for the maxiumum size of 16777216
>>> at
>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>> Source) ~[na:1.7.0_71]
>>> at
>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032
>>> bytes is too large for the maxiumum size of 16777216
>>> at
>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> ... 4 common frames omitted
>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762
>>> CompactionTask.java:136 - Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825
>>> CompactionTask.java:251 - Compacted 1 sstables to
>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,].
>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s.
>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853
>>> CompactionTask.java:136 - Compacting
>>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921
>>> CompactionTask.java:251 - Compacted 1 sstables to
>>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,].
>>> 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s.
>>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532
>>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>>> Thread[SharedPool-Worker-1,5,main]: {}
>>> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too
>>> large for the maxiumum size of 16777216
>>> at
>>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>> Source) ~[na:1.7.0_71]
>>> at
>>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at
>>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>
>>>
>>>
>>> Regards:
>>> Rahul Bhardwaj
>>>
>>> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <
>>> rahul.bhardwaj@indiamart.com> wrote:
>>>
>>>> Hi Joe,
>>>>
>>>> PFA heap dump
>>>>
>>>>
>>>> regards:
>>>> Rahul Bhardwaj
>>>>
>>>>
>>>>
>>>> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <jo...@mac.com> wrote:
>>>>
>>>>> Did you try generating a heap dump so you can look through it to see
>>>>> what’s actually happened?
>>>>>
>>>>>
>>>>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <
>>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> We are using cassandra 2.1 version in a cluster of three machines each
>>>>> with 64 GB RAM
>>>>>
>>>>> The processes are killed by kernel, coz they are eating all memory
>>>>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
>>>>> because we have 64 GB RAM.
>>>>>
>>>>> Please help.
>>>>>
>>>>>
>>>>> Regards:
>>>>> Rahul Bhardwaj
>>>>>
>>>>>
>>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates
>>>>> on this and more: <https://plus.google.com/+indiamart>
>>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>>> Mobile Channel:
>>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>>> <http://m.indiamart.com/>
>>>>>
>>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>>> kyunki Kaam Yahin Banta Hai
>>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on
>>> this and more: <https://plus.google.com/+indiamart>
>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>> Mobile Channel:
>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>> <http://m.indiamart.com/>
>>>
>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
>>> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>
>>>
>>>
>>>
>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on
>> this and more: <https://plus.google.com/+indiamart>
>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>> Mobile Channel:
>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>> <http://m.indiamart.com/>
>>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
>> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>
>>
>>
>
>
> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
> this and more: <https://plus.google.com/+indiamart>
> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
> Mobile Channel:
> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
> <https://play.google.com/store/apps/details?id=com.indiamart.m>
> <http://m.indiamart.com/>
>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>
--
Thanks,
Ryan Svihla
Re: Cassandra consuming whole RAM (64 G)
Posted by Rahul Bhardwaj <ra...@indiamart.com>.
Hi Joe..
Thanks for your valuable solution.. it worked.
But for this problem
*The processes are killed by kernel, coz they are eating all memory
(oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
because we have 64 GB RAM.*
Should I apply patch given for issue
https://issues.apache.org/jira/browse/CASSANDRA-8248 ??
regards:
Rahul Bhardwaj
On Tue, Jan 6, 2015 at 12:42 PM, Joe Ramsey <jo...@mac.com> wrote:
> Thanks Rahul and good luck! I’m really curious to hear what the result is.
>
>
> On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj <ra...@indiamart.com>
> wrote:
>
> Thanks for your response.. i will get back to you with my findings.
>
> On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <jo...@mac.com> wrote:
>
>> That should be “writing too many bytes” not “waiting too many bytes” just
>> for clarity’s sake.
>>
>> On Jan 6, 2015, at 2:03 AM, Joe Ramsey <jo...@mac.com> wrote:
>>
>> I’m not an expert. Really just learning this myself but it looks like
>> according to the stack you’re getting an exception waiting too many bytes
>> to the commit log.
>>
>> That’s controlled by commit_log_segment_size_in_mb setting. The maximum
>> write size that C* will allow is half of the value set for this parameter
>> so if it’s set for 32 (default) that means your max write would be 16MB.
>> (And that’s what’s getting reported in the stack trace. You’re writing
>> 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing the
>> exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and see
>> if the error goes away. It really should get tuned for the amount of
>> writes but that’ll tell you at least if this is the issue.
>>
>> Let me know how it goes!
>>
>> Joe
>>
>>
>>
>> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <ra...@indiamart.com>
>> wrote:
>>
>> Hi Joe,
>>
>> PFB output of system.log
>>
>> tail -n 100 system.log
>> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555
>> CompactionTask.java:251 - Compacted 4 sstables to
>> [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,].
>> 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms =
>> 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge
>> counts were {1:246186, 2:11302, 3:2259, 4:538, }
>> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579
>> CompactionTask.java:251 - Compacted 4 sstables to
>> [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,].
>> 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total
>> partitions merged to 1. Partition merge counts were {1:1, 2:3, }
>> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - Batch
>> of prepared statements for [clickstream.business_feed] is of size 433053,
>> exceeding specified threshold of 5120 by 427933.
>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883
>> CompactionTask.java:136 - Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952
>> CompactionTask.java:251 - Compacted 1 sstables to
>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,].
>> 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s.
>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch
>> of prepared statements for [clickstream.business_feed] is of size 433819,
>> exceeding specified threshold of 5120 by 428699.
>> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982
>> CompactionTask.java:136 - Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
>> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048
>> CompactionTask.java:251 - Compacted 1 sstables to
>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,].
>> 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s.
>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch
>> of prepared statements for [clickstream.business_feed] is of size 436755,
>> exceeding specified threshold of 5120 by 431635.
>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074
>> CompactionTask.java:136 - Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139
>> CompactionTask.java:251 - Compacted 1 sstables to
>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,].
>> 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s.
>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692
>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>> Thread[SharedPool-Worker-1,5,main]: {}
>> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too
>> large for the maxiumum size of 16777216
>> at
>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> Source) ~[na:1.7.0_71]
>> at
>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167
>> CompactionTask.java:136 - Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241
>> CompactionTask.java:251 - Compacted 1 sstables to
>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,].
>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 -
>> ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space:
>> 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
>> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch
>> of prepared statements for [clickstream.im_csl_glusr_daily_activity_count]
>> is of size 60021, exceeding specified threshold of 5120 by 54901.
>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274
>> CompactionTask.java:136 - Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348
>> CompactionTask.java:251 - Compacted 1 sstables to
>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,].
>> 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s.
>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch
>> of prepared statements for [clickstream.business_feed] is of size 517769,
>> exceeding specified threshold of 5120 by 512649.
>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374
>> CompactionTask.java:136 - Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445
>> CompactionTask.java:251 - Compacted 1 sstables to
>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,].
>> 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s.
>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - Batch
>> of prepared statements for [clickstream.im_csl_log] is of size 13220136,
>> exceeding specified threshold of 5120 by 13215016.
>> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092
>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>> Thread[SharedPool-Worker-1,5,main]: {}
>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation
>> of 16800887 bytes is too large for the maxiumum size of 16777216
>> at
>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> Source) ~[na:1.7.0_71]
>> at
>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 bytes
>> is too large for the maxiumum size of 16777216
>> at
>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> ... 4 common frames omitted
>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472
>> CompactionTask.java:136 - Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538
>> CompactionTask.java:251 - Compacted 1 sstables to
>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,].
>> 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s.
>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch
>> of prepared statements for [clickstream.im_csl_log] is of size 13136950,
>> exceeding specified threshold of 5120 by 13131830.
>> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129
>> ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 (24%)
>> on-heap, 0 (0%) off-heap
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325
>> - Writing Memtable-im_csl_log@1246880835(90297505 serialized bytes,
>> 1530359 ops, 20%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364
>> - Completed flushing
>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db
>> (21516225 bytes) for commitlog position
>> ReplayPosition(segmentId=1420516895898, position=16708232)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325
>> - Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840
>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364
>> - Completed flushing
>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db
>> (361277 bytes) for commitlog position
>> ReplayPosition(segmentId=1420516895898, position=16708232)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325
>> - Writing
>> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840
>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364
>> - Completed flushing
>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db
>> (543861 bytes) for commitlog position
>> ReplayPosition(segmentId=1420516895898, position=16708232)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325
>> - Writing
>> Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840
>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364
>> - Completed flushing
>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db
>> (481902 bytes) for commitlog position
>> ReplayPosition(segmentId=1420516895898, position=16708232)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325
>> - Writing Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840
>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364
>> - Completed flushing
>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db
>> (460053 bytes) for commitlog position
>> ReplayPosition(segmentId=1420516895898, position=16708232)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325
>> - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840
>> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364
>> - Completed flushing
>> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db
>> (395084 bytes) for commitlog position
>> ReplayPosition(segmentId=1420516895898, position=16708232)
>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563
>> CompactionTask.java:136 - Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633
>> CompactionTask.java:251 - Compacted 1 sstables to
>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,].
>> 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s.
>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659
>> CompactionTask.java:136 - Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733
>> CompactionTask.java:251 - Compacted 1 sstables to
>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,].
>> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch
>> of prepared statements for [clickstream.im_csl_log] is of size 13480721,
>> exceeding specified threshold of 5120 by 13475601.
>> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537
>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>> Thread[SharedPool-Worker-1,5,main]: {}
>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation
>> of 17138032 bytes is too large for the maxiumum size of 16777216
>> at
>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> Source) ~[na:1.7.0_71]
>> at
>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 bytes
>> is too large for the maxiumum size of 16777216
>> at
>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> ... 4 common frames omitted
>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762
>> CompactionTask.java:136 - Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825
>> CompactionTask.java:251 - Compacted 1 sstables to
>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,].
>> 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s.
>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853
>> CompactionTask.java:136 - Compacting
>> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921
>> CompactionTask.java:251 - Compacted 1 sstables to
>> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,].
>> 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s.
>> 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532
>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>> Thread[SharedPool-Worker-1,5,main]: {}
>> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too
>> large for the maxiumum size of 16777216
>> at
>> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> Source) ~[na:1.7.0_71]
>> at
>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at
>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>
>>
>>
>> Regards:
>> Rahul Bhardwaj
>>
>> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <
>> rahul.bhardwaj@indiamart.com> wrote:
>>
>>> Hi Joe,
>>>
>>> PFA heap dump
>>>
>>>
>>> regards:
>>> Rahul Bhardwaj
>>>
>>>
>>>
>>> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <jo...@mac.com> wrote:
>>>
>>>> Did you try generating a heap dump so you can look through it to see
>>>> what’s actually happened?
>>>>
>>>>
>>>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <
>>>> rahul.bhardwaj@indiamart.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> We are using cassandra 2.1 version in a cluster of three machines each
>>>> with 64 GB RAM
>>>>
>>>> The processes are killed by kernel, coz they are eating all memory
>>>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
>>>> because we have 64 GB RAM.
>>>>
>>>> Please help.
>>>>
>>>>
>>>> Regards:
>>>> Rahul Bhardwaj
>>>>
>>>>
>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on
>>>> this and more: <https://plus.google.com/+indiamart>
>>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>>> Mobile Channel:
>>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>>> <http://m.indiamart.com/>
>>>>
>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART,
>>>> kyunki Kaam Yahin Banta Hai
>>>> <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>>
>>>>
>>>>
>>>
>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on
>> this and more: <https://plus.google.com/+indiamart>
>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>> Mobile Channel:
>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>> <http://m.indiamart.com/>
>>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
>> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>
>>
>>
>>
>
>
> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on
> this and more: <https://plus.google.com/+indiamart>
> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
> Mobile Channel:
> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
> <https://play.google.com/store/apps/details?id=com.indiamart.m>
> <http://m.indiamart.com/>
>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>
>
>
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
答复: Cassandra consuming whole RAM (64 G)
Posted by 李建奇 <li...@jd.com>.
Hi ,
There is a “possible memory leak “ issue with c* 2.1.2. https://issues.apache.org/jira/browse/CASSANDRA-8248
It happen with our c* 2.1.2 cluster . In /proc/{pid}/maps there are a lot of deleted file maps .
_____
Stephen li
发件人: Joe Ramsey [mailto:joe.ramsey@mac.com]
发送时间: 2015年1月6日 15:12
收件人: user@cassandra.apache.org
主题: Re: Cassandra consuming whole RAM (64 G)
Thanks Rahul and good luck! I’m really curious to hear what the result is.
On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj <ra...@indiamart.com> wrote:
Thanks for your response.. i will get back to you with my findings.
On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <jo...@mac.com> wrote:
That should be “writing too many bytes” not “waiting too many bytes” just for clarity’s sake.
On Jan 6, 2015, at 2:03 AM, Joe Ramsey <jo...@mac.com> wrote:
I’m not an expert. Really just learning this myself but it looks like according to the stack you’re getting an exception waiting too many bytes to the commit log.
That’s controlled by commit_log_segment_size_in_mb setting. The maximum write size that C* will allow is half of the value set for this parameter so if it’s set for 32 (default) that means your max write would be 16MB. (And that’s what’s getting reported in the stack trace. You’re writing 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing the exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and see if the error goes away. It really should get tuned for the amount of writes but that’ll tell you at least if this is the issue.
Let me know how it goes!
Joe
On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <ra...@indiamart.com> wrote:
Hi Joe,
PFB output of system.log
tail -n 100 system.log
INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555 CompactionTask.java:251 - Compacted 4 sstables to [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,]. 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms = 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge counts were {1:246186, 2:11302, 3:2259, 4:538, }
INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579 CompactionTask.java:251 - Compacted 4 sstables to [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,]. 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total partitions merged to 1. Partition merge counts were {1:1, 2:3, }
WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 433053, exceeding specified threshold of 5120 by 427933.
INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 433819, exceeding specified threshold of 5120 by 428699.
INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 436755, exceeding specified threshold of 5120 by 431635.
INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 - ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_glusr_daily_activity_count] is of size 60021, exceeding specified threshold of 5120 by 54901.
INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 517769, exceeding specified threshold of 5120 by 512649.
INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13220136, exceeding specified threshold of 5120 by 13215016.
WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
... 4 common frames omitted
INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13136950, exceeding specified threshold of 5120 by 13131830.
INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129 ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 (24%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325 - Writing Memtable-im_csl_log@1246880835(90297505 serialized bytes, 1530359 ops, 20%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db (21516225 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db (361277 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db (543861 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db (481902 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db (460053 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db (395084 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13480721, exceeding specified threshold of 5120 by 13475601.
WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
... 4 common frames omitted
INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too large for the maxiumum size of 16777216
at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
Regards:
Rahul Bhardwaj
On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <ra...@indiamart.com> wrote:
Hi Joe,
PFA heap dump
regards:
Rahul Bhardwaj
On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <jo...@mac.com> wrote:
Did you try generating a heap dump so you can look through it to see what’s actually happened?
On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <ra...@indiamart.com> wrote:
Hi,
We are using cassandra 2.1 version in a cluster of three machines each with 64 GB RAM
The processes are killed by kernel, coz they are eating all memory (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) because we have 64 GB RAM.
Please help.
Regards:
Rahul Bhardwaj
Follow <http://www.indiamart.com/> IndiaMART.com for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU> !!!
Follow <http://www.indiamart.com/> IndiaMART.com for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU> !!!
Follow <http://www.indiamart.com/> IndiaMART.com for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU> !!!
Re: Cassandra consuming whole RAM (64 G)
Posted by Joe Ramsey <jo...@mac.com>.
Thanks Rahul and good luck! I’m really curious to hear what the result is.
> On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj <ra...@indiamart.com> wrote:
>
> Thanks for your response.. i will get back to you with my findings.
>
> On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <joe.ramsey@mac.com <ma...@mac.com>> wrote:
> That should be “writing too many bytes” not “waiting too many bytes” just for clarity’s sake.
>
>> On Jan 6, 2015, at 2:03 AM, Joe Ramsey <joe.ramsey@mac.com <ma...@mac.com>> wrote:
>>
>> I’m not an expert. Really just learning this myself but it looks like according to the stack you’re getting an exception waiting too many bytes to the commit log.
>>
>> That’s controlled by commit_log_segment_size_in_mb setting. The maximum write size that C* will allow is half of the value set for this parameter so if it’s set for 32 (default) that means your max write would be 16MB. (And that’s what’s getting reported in the stack trace. You’re writing 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing the exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and see if the error goes away. It really should get tuned for the amount of writes but that’ll tell you at least if this is the issue.
>>
>> Let me know how it goes!
>>
>> Joe
>>
>>
>>
>>> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com <ma...@indiamart.com>> wrote:
>>>
>>> Hi Joe,
>>>
>>> PFB output of system.log
>>>
>>> tail -n 100 system.log
>>> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555 CompactionTask.java:251 - Compacted 4 sstables to [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,]. 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms = 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge counts were {1:246186, 2:11302, 3:2259, 4:538, }
>>> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579 CompactionTask.java:251 - Compacted 4 sstables to [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,]. 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total partitions merged to 1. Partition merge counts were {1:1, 2:3, }
>>> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 433053, exceeding specified threshold of 5120 by 427933.
>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
>>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 433819, exceeding specified threshold of 5120 by 428699.
>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
>>> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 436755, exceeding specified threshold of 5120 by 431635.
>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
>>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
>>> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too large for the maxiumum size of 16777216
>>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
>>> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
>>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 - ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
>>> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_glusr_daily_activity_count] is of size 60021, exceeding specified threshold of 5120 by 54901.
>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
>>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 517769, exceeding specified threshold of 5120 by 512649.
>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
>>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13220136, exceeding specified threshold of 5120 by 13215016.
>>> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
>>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
>>> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
>>> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
>>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> ... 4 common frames omitted
>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
>>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13136950, exceeding specified threshold of 5120 by 13131830.
>>> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129 ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 (24%) on-heap, 0 (0%) off-heap
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325 - Writing Memtable-im_csl_log@1246880835(90297505 serialized bytes, 1530359 ops, 20%/0% of on/off-heap limit)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db (21516225 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db (361277 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db (543861 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db (481902 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db (460053 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db (395084 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
>>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
>>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13480721, exceeding specified threshold of 5120 by 13475601.
>>> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
>>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
>>> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
>>> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
>>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> ... 4 common frames omitted
>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
>>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
>>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>>> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
>>> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too large for the maxiumum size of 16777216
>>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
>>> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>>
>>>
>>>
>>> Regards:
>>> Rahul Bhardwaj
>>>
>>> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com <ma...@indiamart.com>> wrote:
>>> Hi Joe,
>>>
>>> PFA heap dump
>>>
>>>
>>> regards:
>>> Rahul Bhardwaj
>>>
>>>
>>>
>>> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <joe.ramsey@mac.com <ma...@mac.com>> wrote:
>>> Did you try generating a heap dump so you can look through it to see what’s actually happened?
>>>
>>>
>>>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com <ma...@indiamart.com>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> We are using cassandra 2.1 version in a cluster of three machines each with 64 GB RAM
>>>>
>>>> The processes are killed by kernel, coz they are eating all memory (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) because we have 64 GB RAM.
>>>>
>>>> Please help.
>>>>
>>>>
>>>> Regards:
>>>> Rahul Bhardwaj
>>>>
>>>>
>>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
>>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>
>>>
>>>
>>>
>>>
>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>
>
>
>
>
> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
Re: Cassandra consuming whole RAM (64 G)
Posted by Rahul Bhardwaj <ra...@indiamart.com>.
Thanks for your response.. i will get back to you with my findings.
On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <jo...@mac.com> wrote:
> That should be “writing too many bytes” not “waiting too many bytes” just
> for clarity’s sake.
>
> On Jan 6, 2015, at 2:03 AM, Joe Ramsey <jo...@mac.com> wrote:
>
> I’m not an expert. Really just learning this myself but it looks like
> according to the stack you’re getting an exception waiting too many bytes
> to the commit log.
>
> That’s controlled by commit_log_segment_size_in_mb setting. The maximum
> write size that C* will allow is half of the value set for this parameter
> so if it’s set for 32 (default) that means your max write would be 16MB.
> (And that’s what’s getting reported in the stack trace. You’re writing
> 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing the
> exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and see
> if the error goes away. It really should get tuned for the amount of
> writes but that’ll tell you at least if this is the issue.
>
> Let me know how it goes!
>
> Joe
>
>
>
> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <ra...@indiamart.com>
> wrote:
>
> Hi Joe,
>
> PFB output of system.log
>
> tail -n 100 system.log
> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555
> CompactionTask.java:251 - Compacted 4 sstables to
> [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,].
> 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms =
> 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge
> counts were {1:246186, 2:11302, 3:2259, 4:538, }
> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579
> CompactionTask.java:251 - Compacted 4 sstables to
> [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,].
> 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total
> partitions merged to 1. Partition merge counts were {1:1, 2:3, }
> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - Batch
> of prepared statements for [clickstream.business_feed] is of size 433053,
> exceeding specified threshold of 5120 by 427933.
> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch
> of prepared statements for [clickstream.business_feed] is of size 433819,
> exceeding specified threshold of 5120 by 428699.
> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch
> of prepared statements for [clickstream.business_feed] is of size 436755,
> exceeding specified threshold of 5120 by 431635.
> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too
> large for the maxiumum size of 16777216
> at
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) ~[na:1.7.0_71]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 -
> ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space:
> 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch
> of prepared statements for [clickstream.im_csl_glusr_daily_activity_count]
> is of size 60021, exceeding specified threshold of 5120 by 54901.
> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch
> of prepared statements for [clickstream.business_feed] is of size 517769,
> exceeding specified threshold of 5120 by 512649.
> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - Batch
> of prepared statements for [clickstream.im_csl_log] is of size 13220136,
> exceeding specified threshold of 5120 by 13215016.
> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation
> of 16800887 bytes is too large for the maxiumum size of 16777216
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) ~[na:1.7.0_71]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 bytes
> is too large for the maxiumum size of 16777216
> at
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> ... 4 common frames omitted
> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch
> of prepared statements for [clickstream.im_csl_log] is of size 13136950,
> exceeding specified threshold of 5120 by 13131830.
> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129 ColumnFamilyStore.java:840
> - Enqueuing flush of im_csl_log: 505903980 (24%) on-heap, 0 (0%) off-heap
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325 -
> Writing Memtable-im_csl_log@1246880835(90297505 serialized bytes, 1530359
> ops, 20%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db
> (21516225 bytes) for commitlog position
> ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325 -
> Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840
> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db
> (361277 bytes) for commitlog position
> ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325 -
> Writing
> Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840
> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db
> (543861 bytes) for commitlog position
> ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325 -
> Writing Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840
> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db
> (481902 bytes) for commitlog position
> ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325 -
> Writing Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840
> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db
> (460053 bytes) for commitlog position
> ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325 -
> Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840
> serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364 -
> Completed flushing
> /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db
> (395084 bytes) for commitlog position
> ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch
> of prepared statements for [clickstream.im_csl_log] is of size 13480721,
> exceeding specified threshold of 5120 by 13475601.
> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation
> of 17138032 bytes is too large for the maxiumum size of 16777216
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) ~[na:1.7.0_71]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 bytes
> is too large for the maxiumum size of 16777216
> at
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> ... 4 common frames omitted
> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853
> CompactionTask.java:136 - Compacting
> [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921
> CompactionTask.java:251 - Compacted 1 sstables to
> [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,].
> 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s.
> 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too
> large for the maxiumum size of 16777216
> at
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source) ~[na:1.7.0_71]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>
>
>
> Regards:
> Rahul Bhardwaj
>
> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <
> rahul.bhardwaj@indiamart.com> wrote:
>
>> Hi Joe,
>>
>> PFA heap dump
>>
>>
>> regards:
>> Rahul Bhardwaj
>>
>>
>>
>> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <jo...@mac.com> wrote:
>>
>>> Did you try generating a heap dump so you can look through it to see
>>> what’s actually happened?
>>>
>>>
>>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <
>>> rahul.bhardwaj@indiamart.com> wrote:
>>>
>>> Hi,
>>>
>>> We are using cassandra 2.1 version in a cluster of three machines each
>>> with 64 GB RAM
>>>
>>> The processes are killed by kernel, coz they are eating all memory
>>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
>>> because we have 64 GB RAM.
>>>
>>> Please help.
>>>
>>>
>>> Regards:
>>> Rahul Bhardwaj
>>>
>>>
>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on
>>> this and more: <https://plus.google.com/+indiamart>
>>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>>> Mobile Channel:
>>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>>> <http://m.indiamart.com/>
>>>
>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
>>> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>>
>>>
>>>
>>
>
>
> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on
> this and more: <https://plus.google.com/+indiamart>
> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
> Mobile Channel:
> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
> <https://play.google.com/store/apps/details?id=com.indiamart.m>
> <http://m.indiamart.com/>
>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>
>
>
>
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
Re: Cassandra consuming whole RAM (64 G)
Posted by Joe Ramsey <jo...@mac.com>.
That should be “writing too many bytes” not “waiting too many bytes” just for clarity’s sake.
> On Jan 6, 2015, at 2:03 AM, Joe Ramsey <jo...@mac.com> wrote:
>
> I’m not an expert. Really just learning this myself but it looks like according to the stack you’re getting an exception waiting too many bytes to the commit log.
>
> That’s controlled by commit_log_segment_size_in_mb setting. The maximum write size that C* will allow is half of the value set for this parameter so if it’s set for 32 (default) that means your max write would be 16MB. (And that’s what’s getting reported in the stack trace. You’re writing 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing the exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and see if the error goes away. It really should get tuned for the amount of writes but that’ll tell you at least if this is the issue.
>
> Let me know how it goes!
>
> Joe
>
>
>
>> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com <ma...@indiamart.com>> wrote:
>>
>> Hi Joe,
>>
>> PFB output of system.log
>>
>> tail -n 100 system.log
>> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555 CompactionTask.java:251 - Compacted 4 sstables to [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,]. 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms = 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge counts were {1:246186, 2:11302, 3:2259, 4:538, }
>> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579 CompactionTask.java:251 - Compacted 4 sstables to [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,]. 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total partitions merged to 1. Partition merge counts were {1:1, 2:3, }
>> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 433053, exceeding specified threshold of 5120 by 427933.
>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
>> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 433819, exceeding specified threshold of 5120 by 428699.
>> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
>> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 436755, exceeding specified threshold of 5120 by 431635.
>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
>> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
>> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too large for the maxiumum size of 16777216
>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
>> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
>> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 - ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
>> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_glusr_daily_activity_count] is of size 60021, exceeding specified threshold of 5120 by 54901.
>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
>> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 517769, exceeding specified threshold of 5120 by 512649.
>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
>> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13220136, exceeding specified threshold of 5120 by 13215016.
>> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
>> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
>> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> ... 4 common frames omitted
>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
>> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13136950, exceeding specified threshold of 5120 by 13131830.
>> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129 ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 (24%) on-heap, 0 (0%) off-heap
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325 - Writing Memtable-im_csl_log@1246880835(90297505 serialized bytes, 1530359 ops, 20%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db (21516225 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db (361277 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db (543861 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db (481902 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db (460053 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
>> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db (395084 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
>> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
>> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13480721, exceeding specified threshold of 5120 by 13475601.
>> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
>> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
>> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> ... 4 common frames omitted
>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
>> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
>> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
>> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
>> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too large for the maxiumum size of 16777216
>> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
>> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
>> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>>
>>
>>
>> Regards:
>> Rahul Bhardwaj
>>
>> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com <ma...@indiamart.com>> wrote:
>> Hi Joe,
>>
>> PFA heap dump
>>
>>
>> regards:
>> Rahul Bhardwaj
>>
>>
>>
>> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <joe.ramsey@mac.com <ma...@mac.com>> wrote:
>> Did you try generating a heap dump so you can look through it to see what’s actually happened?
>>
>>
>>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com <ma...@indiamart.com>> wrote:
>>>
>>> Hi,
>>>
>>> We are using cassandra 2.1 version in a cluster of three machines each with 64 GB RAM
>>>
>>> The processes are killed by kernel, coz they are eating all memory (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) because we have 64 GB RAM.
>>>
>>> Please help.
>>>
>>>
>>> Regards:
>>> Rahul Bhardwaj
>>>
>>>
>>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
>>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>
>>
>>
>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>
Re: Cassandra consuming whole RAM (64 G)
Posted by Joe Ramsey <jo...@mac.com>.
I’m not an expert. Really just learning this myself but it looks like according to the stack you’re getting an exception waiting too many bytes to the commit log.
That’s controlled by commit_log_segment_size_in_mb setting. The maximum write size that C* will allow is half of the value set for this parameter so if it’s set for 32 (default) that means your max write would be 16MB. (And that’s what’s getting reported in the stack trace. You’re writing 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing the exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and see if the error goes away. It really should get tuned for the amount of writes but that’ll tell you at least if this is the issue.
Let me know how it goes!
Joe
> On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <ra...@indiamart.com> wrote:
>
> Hi Joe,
>
> PFB output of system.log
>
> tail -n 100 system.log
> INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555 CompactionTask.java:251 - Compacted 4 sstables to [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,]. 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms = 0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge counts were {1:246186, 2:11302, 3:2259, 4:538, }
> INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579 CompactionTask.java:251 - Compacted 4 sstables to [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,]. 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total partitions merged to 1. Partition merge counts were {1:1, 2:3, }
> WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 433053, exceeding specified threshold of 5120 by 427933.
> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
> INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 433819, exceeding specified threshold of 5120 by 428699.
> INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
> INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 436755, exceeding specified threshold of 5120 by 431635.
> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
> INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too large for the maxiumum size of 16777216
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
> INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
> INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 - ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
> WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_glusr_daily_activity_count] is of size 60021, exceeding specified threshold of 5120 by 54901.
> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
> INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch of prepared statements for [clickstream.business_feed] is of size 517769, exceeding specified threshold of 5120 by 512649.
> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
> INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13220136, exceeding specified threshold of 5120 by 13215016.
> WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 bytes is too large for the maxiumum size of 16777216
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> ... 4 common frames omitted
> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
> INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13136950, exceeding specified threshold of 5120 by 13131830.
> INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129 ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 (24%) on-heap, 0 (0%) off-heap
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325 - Writing Memtable-im_csl_log@1246880835(90297505 serialized bytes, 1530359 ops, 20%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db (21516225 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db (361277 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db (543861 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db (481902 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db (460053 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364 - Completed flushing /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db (395084 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)
> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
> INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
> INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch of prepared statements for [clickstream.im_csl_log] is of size 13480721, exceeding specified threshold of 5120 by 13475601.
> WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
> Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 bytes is too large for the maxiumum size of 16777216
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> ... 4 common frames omitted
> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
> INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853 CompactionTask.java:136 - Compacting [SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
> INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921 CompactionTask.java:251 - Compacted 1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,]. 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s. 26 total partitions merged to 26. Partition merge counts were {1:26, }
> WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too large for the maxiumum size of 16777216
> at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]
> at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
>
>
>
> Regards:
> Rahul Bhardwaj
>
> On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com <ma...@indiamart.com>> wrote:
> Hi Joe,
>
> PFA heap dump
>
>
> regards:
> Rahul Bhardwaj
>
>
>
> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <joe.ramsey@mac.com <ma...@mac.com>> wrote:
> Did you try generating a heap dump so you can look through it to see what’s actually happened?
>
>
>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com <ma...@indiamart.com>> wrote:
>>
>> Hi,
>>
>> We are using cassandra 2.1 version in a cluster of three machines each with 64 GB RAM
>>
>> The processes are killed by kernel, coz they are eating all memory (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) because we have 64 GB RAM.
>>
>> Please help.
>>
>>
>> Regards:
>> Rahul Bhardwaj
>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>
>
>
>
>
> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
Re: Cassandra consuming whole RAM (64 G)
Posted by Rahul Bhardwaj <ra...@indiamart.com>.
Hi Joe,
PFB output of system.log
tail -n 100 system.log
INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555
CompactionTask.java:251 - Compacted 4 sstables to
[/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,].
50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms =
0.073765MB/s. 277,719 total partitions merged to 260,285. Partition merge
counts were {1:246186, 2:11302, 3:2259, 4:538, }
INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579
CompactionTask.java:251 - Compacted 4 sstables to
[/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,].
511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s. 7 total
partitions merged to 1. Partition merge counts were {1:1, 2:3, }
WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - Batch
of prepared statements for [clickstream.business_feed] is of size 433053,
exceeding specified threshold of 5120 by 427933.
INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]
INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952
CompactionTask.java:251 - Compacted 1 sstables to
[/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,].
2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s.
26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch of
prepared statements for [clickstream.business_feed] is of size 433819,
exceeding specified threshold of 5120 by 428699.
INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]
INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048
CompactionTask.java:251 - Compacted 1 sstables to
[/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,].
2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s.
26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch of
prepared statements for [clickstream.business_feed] is of size 436755,
exceeding specified threshold of 5120 by 431635.
INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]
INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139
CompactionTask.java:251 - Compacted 1 sstables to
[/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,].
2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s.
26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692
AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-1,5,main]: {}
java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too large
for the maxiumum size of 16777216
at
org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.7.0_71]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]
INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241
CompactionTask.java:251 - Compacted 1 sstables to
[/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,].
2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
26 total partitions merged to 26. Partition merge counts were {1:26, }
INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 -
ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space:
1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936
WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch of
prepared statements for [clickstream.im_csl_glusr_daily_activity_count] is
of size 60021, exceeding specified threshold of 5120 by 54901.
INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]
INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348
CompactionTask.java:251 - Compacted 1 sstables to
[/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,].
2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s.
26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch of
prepared statements for [clickstream.business_feed] is of size 517769,
exceeding specified threshold of 5120 by 512649.
INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]
INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445
CompactionTask.java:251 - Compacted 1 sstables to
[/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,].
2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s.
26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - Batch
of prepared statements for [clickstream.im_csl_log] is of size 13220136,
exceeding specified threshold of 5120 by 13215016.
WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092
AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-1,5,main]: {}
java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of
16800887 bytes is too large for the maxiumum size of 16777216
at
org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.7.0_71]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 bytes
is too large for the maxiumum size of 16777216
at
org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
... 4 common frames omitted
INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]
INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538
CompactionTask.java:251 - Compacted 1 sstables to
[/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,].
2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s.
26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch of
prepared statements for [clickstream.im_csl_log] is of size 13136950,
exceeding specified threshold of 5120 by 13131830.
INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129 ColumnFamilyStore.java:840
- Enqueuing flush of im_csl_log: 505903980 (24%) on-heap, 0 (0%) off-heap
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325 -
Writing Memtable-im_csl_log@1246880835(90297505 serialized bytes, 1530359
ops, 20%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364 -
Completed flushing
/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db
(21516225 bytes) for commitlog position
ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325 -
Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840
serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364 -
Completed flushing
/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db
(361277 bytes) for commitlog position
ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325 -
Writing
Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840
serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364 -
Completed flushing
/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db
(543861 bytes) for commitlog position
ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325 -
Writing Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840
serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364 -
Completed flushing
/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db
(481902 bytes) for commitlog position
ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325 -
Writing Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840
serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364 -
Completed flushing
/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db
(460053 bytes) for commitlog position
ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325 -
Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840
serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)
INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364 -
Completed flushing
/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db
(395084 bytes) for commitlog position
ReplayPosition(segmentId=1420516895898, position=16708232)
INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]
INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633
CompactionTask.java:251 - Compacted 1 sstables to
[/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,].
2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s.
26 total partitions merged to 26. Partition merge counts were {1:26, }
INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]
INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733
CompactionTask.java:251 - Compacted 1 sstables to
[/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,].
2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.
26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch of
prepared statements for [clickstream.im_csl_log] is of size 13480721,
exceeding specified threshold of 5120 by 13475601.
WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537
AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-1,5,main]: {}
java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of
17138032 bytes is too large for the maxiumum size of 16777216
at
org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.7.0_71]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 bytes
is too large for the maxiumum size of 16777216
at
org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
... 4 common frames omitted
INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]
INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825
CompactionTask.java:251 - Compacted 1 sstables to
[/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,].
2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s.
26 total partitions merged to 26. Partition merge counts were {1:26, }
INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853
CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]
INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921
CompactionTask.java:251 - Compacted 1 sstables to
[/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,].
2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s.
26 total partitions merged to 26. Partition merge counts were {1:26, }
WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532
AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-1,5,main]: {}
java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too large
for the maxiumum size of 16777216
at
org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.7.0_71]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]
Regards:
Rahul Bhardwaj
On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <
rahul.bhardwaj@indiamart.com> wrote:
> Hi Joe,
>
> PFA heap dump
>
>
> regards:
> Rahul Bhardwaj
>
>
>
> On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <jo...@mac.com> wrote:
>
>> Did you try generating a heap dump so you can look through it to see
>> what’s actually happened?
>>
>>
>> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <ra...@indiamart.com>
>> wrote:
>>
>> Hi,
>>
>> We are using cassandra 2.1 version in a cluster of three machines each
>> with 64 GB RAM
>>
>> The processes are killed by kernel, coz they are eating all memory
>> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
>> because we have 64 GB RAM.
>>
>> Please help.
>>
>>
>> Regards:
>> Rahul Bhardwaj
>>
>>
>> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on
>> this and more: <https://plus.google.com/+indiamart>
>> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
>> Mobile Channel:
>> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
>> <https://play.google.com/store/apps/details?id=com.indiamart.m>
>> <http://m.indiamart.com/>
>>
>> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
>> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
>> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>>
>>
>>
>
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
Re: Cassandra consuming whole RAM (64 G)
Posted by Rahul Bhardwaj <ra...@indiamart.com>.
Hi Joe,
PFA heap dump
regards:
Rahul Bhardwaj
On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <jo...@mac.com> wrote:
> Did you try generating a heap dump so you can look through it to see
> what’s actually happened?
>
>
> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <ra...@indiamart.com>
> wrote:
>
> Hi,
>
> We are using cassandra 2.1 version in a cluster of three machines each
> with 64 GB RAM
>
> The processes are killed by kernel, coz they are eating all memory
> (oom-killer). We have set JAVA heap to default (i.e. it is using 8G)
> because we have 64 GB RAM.
>
> Please help.
>
>
> Regards:
> Rahul Bhardwaj
>
>
> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on
> this and more: <https://plus.google.com/+indiamart>
> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
> Mobile Channel:
> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
> <https://play.google.com/store/apps/details?id=com.indiamart.m>
> <http://m.indiamart.com/>
>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
> Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
>
>
>
--
Follow IndiaMART.com <http://www.indiamart.com> for latest updates on this
and more: <https://plus.google.com/+indiamart>
<https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile
Channel:
<https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
<https://play.google.com/store/apps/details?id=com.indiamart.m>
<http://m.indiamart.com/>
<https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam
Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!
Re: Cassandra consuming whole RAM (64 G)
Posted by Joe Ramsey <jo...@mac.com>.
Did you try generating a heap dump so you can look through it to see what’s actually happened?
> On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <ra...@indiamart.com> wrote:
>
> Hi,
>
> We are using cassandra 2.1 version in a cluster of three machines each with 64 GB RAM
>
> The processes are killed by kernel, coz they are eating all memory (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) because we have 64 GB RAM.
>
> Please help.
>
>
> Regards:
> Rahul Bhardwaj
>
>
> Follow IndiaMART.com <http://www.indiamart.com/> for latest updates on this and more: <https://plus.google.com/+indiamart> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART> Mobile Channel: <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8> <https://play.google.com/store/apps/details?id=com.indiamart.m> <http://m.indiamart.com/>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU>!!!