[jira] [Commented] (CASSANDRA-12662) OOM when using SASI index

["DOAN DuyHai (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15501440#comment-15501440 ] 

DOAN DuyHai commented on CASSANDRA-12662:
-----------------------------------------

Default hardcoded value for memIndexTable is 1Gb: https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java#L369-L372

bq. 2.8Gb of the heap is taken by the index data, pending for flush (see the screenshot)

When you have more than 1Gb of index data, SASI flushes the index by chunks of 1Gb into temporary index files : 
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java#L247-L250

Then it needs a 2nd pass to merge them into memory to write the final index file, see here:  https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/index/sasi/disk/PerSSTableIndexWriter.java#L311-L326

Thus, it may take a while to write the final SASI index file. The speed of all this depends on many factors, mainly CPU and Disk I/O. 

What are you disk hardware specs ? SSD ? Spinning disk ? shared storage ?

bq. Why can't Cassandra keep up with the inserted data and flush it?

Write are CPU-intensive. Compactions are more disk I/O intensive

bq. What resources/configuration should be changed to improve the performance?

Right now, 4 cores CPU is below the official recommendation to run Cassandra in production, which is 8 cores CPU. Same for RAM, recommendation is 32Gb, see here: http://cassandra.apache.org/doc/latest/operating/hardware.html


> OOM when using SASI index
> -------------------------
>
>                 Key: CASSANDRA-12662
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12662
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Linux, 4 CPU cores, 16Gb RAM, Cassandra process utilizes ~8Gb, of which ~4Gb is Java heap
>            Reporter: Maxim Podkolzine
>            Priority: Critical
>             Fix For: 3.6
>
>         Attachments: memory-dump.png
>
>
> 2.8Gb of the heap is taken by the index data, pending for flush (see the screenshot). As a result the node fails with OOM.
> Questions:
> - Why can't Cassandra keep up with the inserted data and flush it?
> - What resources/configuration should be changed to improve the performance?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)