You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Scott Carey (Jira)" <ji...@apache.org> on 2020/11/15 03:16:00 UTC
[jira] [Comment Edited] (CASSANDRA-16071)
max_compaction_flush_memory_in_mb is interpreted as bytes
[ https://issues.apache.org/jira/browse/CASSANDRA-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17232194#comment-17232194 ]
Scott Carey edited comment on CASSANDRA-16071 at 11/15/20, 3:15 AM:
--------------------------------------------------------------------
Its worse than I remembered. Updating to the latest cassandra with this fix in will take down nodes. The configuration option is on the index itself, and there is no 'alter index' option to reset the max_compaction_flush_memory_in_mb value. The whole cluster must be updated to the new version, then compaction has to be either stopped or delicately monitored, then _new_ indexes need to be built with the updated values for max_compaction_flush_memory_in_mb, and then the old indexes can be removed.
The new values will destroy old versions with death-by-a-million-open-files, and the old values will destroy new nodes with OOM during compaction.
I wish I had seen this fix before it was committed. It should have detected 'unusually large' values and interpreted them as bytes.
For example, '20480000' is 20MB if interpreted as bytes. If interpreted as MB its way over any reasonable size – 20TB. A safe bet would be to interpret anything over 100000 (100GB in MB, but only 100K if bytes) as bytes, and anything below that as MB. Then this wouldn't be a migration nightmare.
was (Author: scott_carey):
Its worse than I remembered. Updating to the latest cassandra with this fix in will take down nodes. The configuration option is on the index itself, and there is no 'alter index' option to reset the max_compaction_flush_memory_in_mb value. The whole cluster must be updated to the new version, then compaction has to be either stopped or delicately monitored, then _new_ indexes need to be built with the updated values for max_compaction_flush_memory_in_mb, and then the old indexes can be removed.
The new values will destroy old versions with death-by-a-million-open-files, and the old values will destroy new nodes with OOM during compaction.
I wish I had seen this fix before it was committed. It should have detected 'unusually large' values and interpreted them as bytes.
For example, '20480000' is 20MB if interpreted as bytes. If interpreted as MB its way over any reasonable size – 20TB. A safe bet would be to interpret anything over 100000 (100GB in MB, but only 100K if bytes) as bytes, and anything below that as GB. Then this wouldn't be a migration nightmare.
> max_compaction_flush_memory_in_mb is interpreted as bytes
> ---------------------------------------------------------
>
> Key: CASSANDRA-16071
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16071
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/SASI
> Reporter: Michael Semb Wever
> Assignee: Michael Semb Wever
> Priority: Normal
> Fix For: 4.0, 3.11.8, 4.0-beta2
>
>
> In CASSANDRA-12662, [~scottcarey] [reported|https://issues.apache.org/jira/browse/CASSANDRA-12662?focusedCommentId=17070055&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17070055] that the {{max_compaction_flush_memory_in_mb}} setting gets incorrectly interpreted in bytes rather than megabytes as its name implies.
> {quote}
> 1. the setting 'max_compaction_flush_memory_in_mb' is a misnomer, it is actually memory in BYTES. If you take it at face value, and set it to say, '512' thinking that means 512MB, you will produce a million temp files rather quickly in a large compaction, which will exhaust even large values of max_map_count rapidly, and get the OOM: Map Error issue above and possibly have a very difficult situation to get a cluster back into a place where nodes aren't crashing while initilaizing or soon after. This issue is minor if you know about it in advance and set the value IN BYTES.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org