[jira] [Comment Edited] (FLINK-24401) TM cannot exit after Metaspace OOM

["Piotr Nowojski (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-24401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17430011#comment-17430011 ] 

Piotr Nowojski edited comment on FLINK-24401 at 10/19/21, 7:02 AM:
-------------------------------------------------------------------

[~fanrui], can you post the actual second's OOM stack trace?

As the error was caused by {{org.apache.flink.runtime.taskexecutor.TaskManagerRunner.Result}}, I wonder if we should actually just treat all OOMs the sam way, regardless if it's meta space or not. [~trohrmann], what do you think? I'm asking as you were doing the review of that change, and I don't know the motivation behind it. Was it just best effort thing that we added, just because we thought we could? Or was it addressing some problem that users were complaining about?
{quote}
In case of Metaspace OOM error, we try a graceful TM shutdown to notify JM because it is not expected to require class loading for that and cause further failures.
{quote}
After all this assumption is clearly in the wrong.


was (Author: pnowojski):
[~fanrui], can you post the actual second's OOM stack trace?

As the error was caused by {{org.apache.flink.runtime.taskexecutor.TaskManagerRunner.Result}}, I wonder if we should actually just treat all OOMs the sam way, regardless if it's meta space or not. [~trohrmann], what do you think? I'm asking as you were doing the review of that change, and I don't know the motivation behind it. Was it just best effort thing that we added, just because we thought we could?
{quote}
In case of Metaspace OOM error, we try a graceful TM shutdown to notify JM because it is not expected to require class loading for that and cause further failures.
{quote}
After all this assumption is clearly in the wrong.

> TM cannot exit after Metaspace OOM
> ----------------------------------
>
>                 Key: FLINK-24401
>                 URL: https://issues.apache.org/jira/browse/FLINK-24401
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Coordination, Runtime / Task
>    Affects Versions: 1.12.0, 1.13.0
>            Reporter: future
>            Priority: Major
>             Fix For: 1.14.1, 1.13.4
>
>         Attachments: image-2021-09-29-12-00-28-510.png, image-2021-09-29-12-00-44-812.png
>
>
> Hi masters, from the code and log, we can see that OOM will terminateJVM directly, but Metaspace OutOfMemoryError will graceful shutdown. The code comment mentions: {{_it does not usually require more class loading to fail again with the Metaspace OutOfMemoryError_.}}.
> But we encountered: after Metaspace OutOfMemoryError, {{_java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.runtime.taskexecutor.TaskManagerRunner$Result_.}}, makes Tm unable to exit, keeps trying again, keeps NoClassDefFoundError, keeps class loading failure, until kill tm by manually.
> I want to add a catch Throwable in the onFatalError method, and directly terminateJVM() in the catch. Is there any problem with this strategy? 
>  
> [code link |https://github.com/apache/flink/blob/4fe9f525a92319acc1e3434bebed601306f7a16f/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerRunner.java#L312]
> picture:
>  
> !image-2021-09-29-12-00-44-812.png|width=1337,height=692!
>   !image-2021-09-29-12-00-28-510.png!
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)