You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Pawel Bartoszek (JIRA)" <ji...@apache.org> on 2019/07/29 14:29:00 UTC
[jira] [Updated] (FLINK-13472) taskmanager.jvm-exit-on-oom doesn't
work reliably with YARN
[ https://issues.apache.org/jira/browse/FLINK-13472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pawel Bartoszek updated FLINK-13472:
------------------------------------
Description:
I have added *taskmanager.jvm-exit-on-oom* flag to the task manager starting arguments. During my testing (simulating oom) I noticed that sometimes YARN containers were still in RUNNING state even though they should haven been killed on OutOfMemory errors with the flag on.
I could find RUNNING containers with the last log lines like this.
{code:java}
2019-07-26 13:32:51,396 ERROR org.apache.flink.runtime.taskmanager.Task - Encountered fatal error java.lang.OutOfMemoryError - terminating the JVM
java.lang.OutOfMemoryError: Metaspace
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369){code}
Does YARN make it tricky to forcefully kill JVM after OutOfMemory error?
*Workaround*
When using -XX:+ExitOnOutOfMemoryError JVM flag containers get always terminated!
was:
I have added *taskmanager.jvm-exit-on-oom* flag to the task manager starting arguments. During my testing (simulating oom) I noticed that sometimes YARN containers were still in RUNNING state even though they should haven been killed on OutOfMemory errors with the flag on.
I could find RUNNING containers with the last log lines like this.
{code:java}
2019-07-26 13:32:51,396 ERROR org.apache.flink.runtime.taskmanager.Task - Encountered fatal error java.lang.OutOfMemoryError - terminating the JVM
java.lang.OutOfMemoryError: Metaspace
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369){code}
Does YARN make it tricky to forcefully kill JVM after OutOfMemory error?
> taskmanager.jvm-exit-on-oom doesn't work reliably with YARN
> -----------------------------------------------------------
>
> Key: FLINK-13472
> URL: https://issues.apache.org/jira/browse/FLINK-13472
> Project: Flink
> Issue Type: Bug
> Components: core
> Affects Versions: 1.6.3
> Reporter: Pawel Bartoszek
> Priority: Major
>
> I have added *taskmanager.jvm-exit-on-oom* flag to the task manager starting arguments. During my testing (simulating oom) I noticed that sometimes YARN containers were still in RUNNING state even though they should haven been killed on OutOfMemory errors with the flag on.
> I could find RUNNING containers with the last log lines like this.
> {code:java}
> 2019-07-26 13:32:51,396 ERROR org.apache.flink.runtime.taskmanager.Task - Encountered fatal error java.lang.OutOfMemoryError - terminating the JVM
> java.lang.OutOfMemoryError: Metaspace
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369){code}
>
> Does YARN make it tricky to forcefully kill JVM after OutOfMemory error?
>
> *Workaround*
>
> When using -XX:+ExitOnOutOfMemoryError JVM flag containers get always terminated!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)