You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Andrey Zagrebin (Jira)" <ji...@apache.org> on 2020/03/11 19:54:00 UTC

[jira] [Comment Edited] (FLINK-16225) Metaspace Out Of Memory should be handled as Fatal Error in TaskManager

    [ https://issues.apache.org/jira/browse/FLINK-16225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057253#comment-17057253 ] 

Andrey Zagrebin edited comment on FLINK-16225 at 3/11/20, 7:53 PM:
-------------------------------------------------------------------

[~sewen]

I have not found any error handling in ClassLoader out of the box. So I suppose you mean that e.g. our ChildFirstClassLoader and ParentFirstClassLoader should extend one more base ClassLoader class. This class can wrap actual class loading with catching the OutOfMemoryError and calling the handler. As I understand, this has to be done to share the user ClassLoader with any child threads created from the task thread instead of just catching it in the task thread, right?

Although this should work for most of the cases, just wondering about any other threads and their error handling. As I understand, if the Metaspace limit is close to be reached in JVM process then any class loading can potentially cause the OutOfMemoryError, also any Flink classes unused so far and suddenly loaded. This should be a rare case. Unification of error handling in all threads in Flink would be of course quite a big effort.

Also, do you think we should try the graceful cancelation for `OutOfMemoryError: Metaspace`? At the moment, this is not the case in TaskManagerRunner#onFatalError. Although, it is more likely that cancelation succeeds for this type of OutOfMemoryError comparing to heap OOM.


was (Author: azagrebin):
[~sewen]

I have not found any error handling in ClassLoader out of the box. So I suppose you mean that e.g. our ChildFirstClassLoader and ParentFirstClassLoader should extend one more base ClassLoader class. This class can wrap actual class loading with catching the OutOfMemoryError and calling the handler. As I understand, this has to be done to share the user ClassLoader with any child threads created from the task thread instead of just catching it in the task thread, right?

Although this should work for most of the cases, just wondering about any other threads and their error handling. As I understand, if the Metaspace limit is close to be reached in JVM process then any class loading can potentially cause the OutOfMemoryError, also any Flink classes unused so far and suddenly loaded. This should be a rare case. Unification of error handling in all threads in Flink would be of course quite a big effort.

> Metaspace Out Of Memory should be handled as Fatal Error in TaskManager
> -----------------------------------------------------------------------
>
>                 Key: FLINK-16225
>                 URL: https://issues.apache.org/jira/browse/FLINK-16225
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Task
>    Affects Versions: 1.10.0
>            Reporter: Stephan Ewen
>            Assignee: Andrey Zagrebin
>            Priority: Critical
>              Labels: usability
>             Fix For: 1.10.1, 1.11.0
>
>
> When an {{OutOfMemory (Metaspace)}} exception happens, there is usually no way to recover. This is often the result of user code or libraries that have subtle class loading leaks.
> The one way to recover is to kill the TaskManagers and to let the resource orchestrators (K8s, Yarn, Mesos) restart them. Flink's fault tolerance should then be able to recover the job.
> I would suggest to implement this the following way:
> * The user code ClassLoader takes an "OOM Handler", which is called when class loading causes an OOM exception.
> * The handler wraps this into an Exception with a good error message (see below) and invokes the TaskManager's {{FatalErrorHandler}}.
> * The {{FatalErrorHandler}} in turn should attempt to cancel everything and notify the JM before shutting down. That way, we get decent error reporting and users can see what is going on.
> The error message should describe the following:
> * If user sees the error consistently on the first deploy, then the metaspace is simply too small for their application, and they need to explicitly increase it in the configuration
> * If the user sees occasionally TaskManagers in a session cluster failing with that exception when deploying new jobs, then some user code or library probably has a class leak. The TM failure / restart is done in order to forcefully clean up.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)