You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Isaac Philip Liu (Jira)" <ji...@apache.org> on 2026/10/17 07:10:00 UTC
[jira] [Updated] (CASSANDRA-19831) Improve JVMStabilityInspector error message to clarify off-heap vs heap OOM errors
[ https://issues.apache.org/jira/browse/CASSANDRA-19831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Isaac Philip Liu updated CASSANDRA-19831:
-----------------------------------------
Status: In Progress (was: Patch Available)
> Improve JVMStabilityInspector error message to clarify off-heap vs heap OOM errors
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-19831
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19831
> Project: Apache Cassandra
> Issue Type: Task
> Components: Local/Other
> Reporter: Brad Schoening
> Assignee: Isaac Philip Liu
> Priority: Low
> Attachments: ci_summary.html, results_details.tar.xz
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> When we see Direct buffer memory errors, the JVMStabilityInspector appears to suggest it's a heap OOM to casual users. This can confuse users who think that increasing the heap may help solve the problem. A vendors support staff incorrectly suggested increasing the heap in this scenario.
> In inspectThrowable(), when t is a non-heap OOM (e.g. Direct buffer memory), forceHeapSpaceOomMaybe(t) throws a brand-new JVM-generated OutOfMemoryError: Java heap space, and that error propagates up the stack. The last, most prominent stack trace the user sees says Java heap space, with a stack pointing into forceHeapSpaceOomMaybe — not into Bits.reserveMemory. The synthetic OOM doesn't have a direct reference to the original error. The real cause is only findable by scrolling up in the log to the terse "Force heap space OutOfMemoryError in the presence of" line.
> A more clear error message could be something like:
> Off-heap OOM forcing heap space OutOfMemoryError after OutOfMemoryError: Direct buffer memory
> {quote}2024-08-01 17:00:00,16 [ERROR] [Messaging-EventLoop-3-6] cluster_id=001 ip_address=172.0.0.1 JVMStabilityInspector.java:151 - Force heap space OutOfMemoryError in the presence of
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
> at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
> at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318)
> at org.apache.cassandra.utils.MerkleTree.allocate(MerkleTree.java:742)
> at org.apache.cassandra.utils.MerkleTree.deserializeOffHeap(MerkleTree.java:780)
> {quote}
> The comment in the code is clear, but
> {quote} /**
> * Intentionally produce a heap space OOM upon seeing a non heap memory OOM.
> * Direct buffer OOM cannot trigger JVM OOM error related options,
> * e.g. OnOutOfMemoryError, HeapDumpOnOutOfMemoryError, etc.
> * See CASSANDRA-15214 and CASSANDRA-17128 for more details
> */
> {quote}
> but the log message users see begins with 'Force heap space OutOfMemoryError', which suggests a heap memory issue, not off-heap memory. Very confusing for general users and junior support staff.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org