[jira] Commented: (CASSANDRA-1177) OutOfMemory on heavy inserts

["Torsten Curdt (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884741#action_12884741 ] 

Torsten Curdt commented on CASSANDRA-1177:
------------------------------------------

Some more information:

We had an old-ish JDK in place. Upgrading to the latest helped but did not fix it.
But we are still seeing those problems for un-throttled inserts through THRIFT.

 - replication factor 3
 - consistency level ONE
 - 6 nodes involved
 - 4 workers at 200-300 writes/s
 - writes are about 300 bytes in size

We switched to use the low level StorageProxy for the bulk import.
Using that the cluster behaved just beautiful. No problems there.
Much much faster!

So I assume as long as we don't insert too fast we should be OK.
But that it's quite a scary situation if the ring does not recover properly.
...and 1200 writes/s for a 6 node cluster is not really that much.

> OutOfMemory on heavy inserts
> ----------------------------
>
>                 Key: CASSANDRA-1177
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1177
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.6.2
>         Environment: SunOS 5.10, x86 32bit, Jave Hotspot Server VM 11.2-b01 mixed mode
> Sun SDK 1.6.0_12-b04
>            Reporter: Torsten Curdt
>            Priority: Critical
>         Attachments: bug report.zip, commitlog.txt, data.txt
>
>
> We have cluster of 6 Cassandra 0.6.2 nodes running under SunOS (see environment).
> On initial import (using the thrift API) we see some weird behavior of half the cluster. While cas04-06 look fine as you can see from the attached munin graphs, the other 3 nodes kept on GCing (see log file) until they became unreachable and went OOM. (This is also why the stats are so spotty - munin could no longer reach the boxes) We have seen the same behavior on 0.6.2 and 0.6.1. This started after around 100 million inserts.
> Looking at the hprof (which is of course to big to attach) we see lots of ConcurrentSkipListMap$Node's and quite some Column objects. Please see the stats attached.
> This looks similar to https://issues.apache.org/jira/browse/CASSANDRA-1014 but we are not sure it really is the same.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.