You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Scott Carey (Jira)" <ji...@apache.org> on 2020/03/28 22:26:00 UTC
[jira] [Comment Edited] (CASSANDRA-12662) OOM when using SASI index
[ https://issues.apache.org/jira/browse/CASSANDRA-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17070055#comment-17070055 ]
Scott Carey edited comment on CASSANDRA-12662 at 3/28/20, 10:25 PM:
--------------------------------------------------------------------
I have had a similar issue, with SASI using unbounded memory during compaction. I have done quite a bit of debugging and ran into various things that possibly should be separate issues:
1. the setting 'max_compaction_flush_memory_in_mb' is a misnomer, it is actually memory in BYTES. If you take it at face value, and set it to say, '512' thinking that means 512MB, you will produce a million temp files rather quickly in a large compaction, which will exhaust even large values of max_map_count rapidly, and get the OOM: Map Error issue above and possibly have a very difficult situation to get a cluster back into a place where nodes aren't crashing while initilaizing or soon after. This issue is minor if you know about it in advance and set the value IN BYTES.
2. The setting 'max_compaction_flush_memory_in_mb' doesn't work at all if the compaction is outputting more than one file – so any large compaction with LCS is especially bad. Instead, this setting controls the max memory used PER OUTPUT FILE. WIth 1000 output SSTable files, a TB of Java heap could be eaten up... not that anyone runs with a cassandra node with a TB java heap. The fix here seems to be simple: When the compaction ends one file and starts the next, the SASI index should be flushed, or ideally finalized for that particulare Data file. (Note, even an 'ordinary' LCS compaction outputs approximately 10 SSTables by default, but there are many times when it may choose a larger swath of data to compact, and then there is of course a major compaction).
A few more SASI bugs, (sorry for not filing issues):
3. A query that uses a SASI index will fail to filter on any partition keys. So if you have a table with the key like ((subkey1, subkey2), clusterkey1) and perhaps index on static column 'scol1', then a query that has a WHERE clause similar to [ scol1 = 'something' and subkey2 = 'abc' ] (WITH FILTERING of course) then the query will return rows where subkey2 is not 'abc'.
4. If two SASI indexes are eligible, there is no rhyme or reason for which one is used for the query.... well there is, the 'oldest' index seems to be preferred, as the optimizer doesn't seem to know anything about the cardinality of the indexes.
#2 above is pure awful and seems related to this bug, which is why I'm commenting here. It is probably the simplest fix, if all that is done is flushing when a file completes without any optimization, and IMO is worthy of a fix in the 3.11.x branch.
#1 is annoying but has a workaround.
#3 is problematic, but one can filter on the client side and I assume the fix is probably more tricky. If its trivial, then maybe it belongs in 3.11.x, that would be nice but I suspect its not a simple fix.
#4 is most certainly something for the larger update to SASI, I doubt there is any quick, safe fix for the 3.11.x branch.
was (Author: scottcarey):
I have had a similar issue, with SASI using unbounded memory during compaction. I have done quite a bit of debugging and ran into various things that possibly should be separate issues:
1. the setting 'max_compaction_flush_memory_in_mb' is a misnomer, it is actually memory in BYTES. If you take it at face value, and set it to say, '512' you will produce a million temp files rather quickly in a large compaction, which will exhaust even large values of max_map_count rapidly, and get the OOM: Map Error issue above and possibly have a very difficult situation to get a cluster back into a place where nodes aren't crashing while initilaizing or soon after. This issue is minor if you know about it in advance and set the value IN BYTES.
2. The setting 'max_compaction_flush_memory_in_mb' doesn't work at all if the compaction is outputting more than one file – so any large compaction with LCS is especially bad. Instead, this setting controls the max memory used PER OUTPUT FILE. WIth 1000 output SSTable files, a TB of Java heap could be eaten up... not that anyone runs with a cassandra node with a TB java heap. The fix here seems to be simple: When the compaction ends one file and starts the next, the SASI index should be flushed, or ideally finalized for that particulare Data file. (Note, even an 'ordinary' LCS compaction outputs approximately 10 SSTables by default, but there are many times when it may choose a larger swath of data to compact, and then there is of course a major compaction).
A few more SASI bugs, (sorry for not filing issues):
3. A query that uses a SASI index will fail to filter on any partition keys. So if you have a table with the key like ((subkey1, subkey2), clusterkey1) and perhaps index on static column 'scol1', then a query that has a WHERE clause similar to [ scol1 = 'something' and subkey2 = 'abc' ] (WITH FILTERING of course) then the query will return rows where subkey2 is not 'abc'.
4. If two SASI indexes are eligible, there is no rhyme or reason for which one is used for the query.... well there is, the 'oldest' index seems to be preferred, as the optimizer doesn't seem to know anything about the cardinality of the indexes.
#2 above is pure awful and seems related to this bug, which is why I'm commenting here. It is probably the simplest fix, if all that is done is flushing when a file completes without any optimization, and IMO is worthy of a fix in the 3.11.x branch.
#1 is annoying but has a workaround.
#3 is problematic, but one can filter on the client side and I assume the fix is probably more tricky. If its trivial, then maybe it belongs in 3.11.x, that would be nice but I suspect its not a simple fix.
#4 is most certainly something for the larger update to SASI, I doubt there is any quick, safe fix for the 3.11.x branch.
> OOM when using SASI index
> -------------------------
>
> Key: CASSANDRA-12662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12662
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux, 4 CPU cores, 16Gb RAM, Cassandra process utilizes ~8Gb, of which ~4Gb is Java heap
> Reporter: Maxim Podkolzine
> Priority: Urgent
> Fix For: 3.11.x
>
> Attachments: memory-dump.png
>
>
> 2.8Gb of the heap is taken by the index data, pending for flush (see the screenshot). As a result the node fails with OOM.
> Questions:
> - Why can't Cassandra keep up with the inserted data and flush it?
> - What resources/configuration should be changed to improve the performance?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org