You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Bo Cui (Jira)" <ji...@apache.org> on 2021/05/10 06:11:00 UTC
[jira] [Comment Edited] (FLINK-9844) PackagedProgram does not close
URLClassLoader
[ https://issues.apache.org/jira/browse/FLINK-9844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17341679#comment-17341679 ]
Bo Cui edited comment on FLINK-9844 at 5/10/21, 6:10 AM:
---------------------------------------------------------
hi [~chesnay]
but after the patch, `flink run -m yarn-cluster -d examples/streaming/WindowJoin.jar` throws exception
{code:java}
Exception in thread "Thread-6" java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'.Exception in thread "Thread-6" java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'. at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:164) at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResource(FlinkUserCodeClassLoaders.java:183) at org.apache.hadoop.conf.Configuration.getResource(Configuration.java:2831) at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3279) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3238) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:3071) at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2915) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2897) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214) at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1802) at org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:182) at org.apache.hadoop.util.ShutdownHookManager.shutdownExecutor(ShutdownHookManager.java:143) at org.apache.hadoop.util.ShutdownHookManager.access$300(ShutdownHookManager.java:64) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:100){code}
Did you consider detached mode
was (Author: bo cui):
hi [~chesnay]
but after the patch, `flink run -m yarn-cluster -d examples/streaming/WindowJoin.jar` throws exception
Exception in thread "Thread-6" java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'.Exception in thread "Thread-6" java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'. at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:164) at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResource(FlinkUserCodeClassLoaders.java:183) at org.apache.hadoop.conf.Configuration.getResource(Configuration.java:2831) at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3279) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3238) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:3071) at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2915) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2897) at org.apache.hadoop.conf.Configuration.get(Configuration.java:1214) at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1802) at org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:182) at org.apache.hadoop.util.ShutdownHookManager.shutdownExecutor(ShutdownHookManager.java:143) at org.apache.hadoop.util.ShutdownHookManager.access$300(ShutdownHookManager.java:64) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:100)
> PackagedProgram does not close URLClassLoader
> ---------------------------------------------
>
> Key: FLINK-9844
> URL: https://issues.apache.org/jira/browse/FLINK-9844
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client, Runtime / REST
> Affects Versions: 1.4.2, 1.5.1, 1.6.0
> Reporter: Chesnay Schepler
> Assignee: Chesnay Schepler
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.4, 1.12.2, 1.13.0
>
>
> The {{PackagedProgram}} class creates a user-code classloader to execute the programs {{main}} method.
> This classloader is a {{URLClassLoader}} (except in the case of some tests), which contains opened {{JarFiles}} for all accessed jars.
> The {{URLClassLoader}} class implements {{Closeable}} for the purpose of cleaning up closeable resources, however we never actually call this method. The {{PackagedProgram}} only works against the {{ClassLoader}} class, which doesn't implement {{Closeable}}.
> As a result, deleting a jar after submitting it with the {{JarRunHandler}} currently fails on Windows, since the jar is still in use.
> As far as i can tell after the JobGraph has been compiled this classloader is no longer necessary and should be closed. (This assumes that a given {{PackagedProgram}} is only used once to create a JobGraph}}, otherwise we'd needa dedicate close method)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)