You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Puneet Duggal (Jira)" <ji...@apache.org> on 2021/09/13 15:59:00 UTC

[jira] [Comment Edited] (FLINK-20333) Flink standalone cluster throws metaspace OOM after submitting multiple PyFlink UDF jobs.

    [ https://issues.apache.org/jira/browse/FLINK-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17414278#comment-17414278 ] 

Puneet Duggal edited comment on FLINK-20333 at 9/13/21, 3:58 PM:
-----------------------------------------------------------------

Hi, Is this issue resolved for java client... Asking this because in prod task manager with 256Mb jvm metaspace... i am getting metaspace OOM by deploying 4-5 jobs on single task manager irrespective of their parallelism. I am using flink 1.12.0 version. [~dian.fu]


was (Author: pd17):
Hi, Is this issue resolved for java client... Asking this because in prod task manager with 256Mb jvm metaspace... i am getting metaspace OOM by deploying 4-5 jobs on single task manager irrespective of their parallelism. I am using flink 1.12.0 version.

> Flink standalone cluster throws metaspace OOM after submitting multiple PyFlink UDF jobs.
> -----------------------------------------------------------------------------------------
>
>                 Key: FLINK-20333
>                 URL: https://issues.apache.org/jira/browse/FLINK-20333
>             Project: Flink
>          Issue Type: Bug
>          Components: API / Python
>            Reporter: Wei Zhong
>            Assignee: Wei Zhong
>            Priority: Critical
>              Labels: pull-request-available
>             Fix For: 1.11.3, 1.12.0
>
>
> Currently the Flink standalone cluster will throw metaspace OOM after submitting multiple PyFlink UDF jobs. The root cause is that currently the PyFlink classes are running in user classloader and so each job creates a separate user class loader to load PyFlink related classes. There are many soft references and Finalizers in memory (introduced by the underlying Netty), which prevents the garbage collection of the user classloader of already finished PyFlink jobs. 
> Due to their existence, it needs multiple full gc to reclaim the classloader of the completed job. If only one full gc is performed before the metaspace space is insufficient, then OOM will occur.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)