[jira] [Commented] (CASSANDRA-16939) Shutdown hook is removed after non-fatal OutOfMemoryError

["Dmitry Konstantinov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18068583#comment-18068583 ] 

Dmitry Konstantinov commented on CASSANDRA-16939:
-------------------------------------------------

aren't we trigger a heap OOM in case of a non-heap OOM here? org.apache.cassandra.utils.JVMStabilityInspector#forceHeapSpaceOomMaybe

> Shutdown hook is removed after non-fatal OutOfMemoryError
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-16939
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-16939
>             Project: Apache Cassandra
>          Issue Type: Bug
>          Components: Legacy/Core
>            Reporter: Paulo Motta
>            Assignee: shanizta siddiqua
>            Priority: Normal
>              Labels: lhf
>             Fix For: 6.x
>
>          Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> On CASSANDRA-7507 the drain shutdown hook is removed when the process hits an OutOfMemoryError, to avoid trying to do a clean shutdown when the node runs out of heap space.
> On CASSANDRA-13006 cassandra started relying on JVM flags (ExitOnOutOfMemoryError and CrashOnOutOfMemoryError) to stop the node when hitting and OutOfMemoryError.
> However, there are non-fatal {{OutOfMemoryErrors}} such as {{OutOfMemory: unable to create new native thread}} or {{OutOfMemory: map failed}} which [do not cause the process to crash even with the ExitOnOutOfMemoryError or CrashOnOutOfMemoryError flags|https://bugs.openjdk.java.net/browse/JDK-8155004].
> Since the shutdown hook is removed after non-fatal OutOfMemory errors, it's no longer possible to do a clean shutdown (via {{SIGTERM kill}} or {{nodetool stopdaemon}}).
> I believe the intent of CASSANDRA-7507 was to remove the shutdown hook only on fatal OutOfMemoryErrors (such as Heap Space Exhausted), those causing the node to crash. If a node is kept running after an OutOfMemoryError, this should not prevent the node from being cleanly shutdown afterwards.
> We should either make the JVM exit on any OutOfMemory error, or remove the drain shutdown hook only on fatal OutOfMemoryErrors, those that will cause the JVM to crash straight away.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org