You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/12/21 06:17:58 UTC
[GitHub] [iceberg] yittg edited a comment on issue #3776: Flink connector thrown `Trying to access closed classloader`
yittg edited a comment on issue #3776:
URL: https://github.com/apache/iceberg/issues/3776#issuecomment-998502908
After some digging, i find the direct cause is [that](https://github.com/apache/iceberg/blob/5009949ba4377ac5a8572ff7ae70e886c9e33bec/core/src/main/java/org/apache/iceberg/ManifestReader.java?#L100-L102):
when we build a `AvroIterable`, we didn't set the class loader, then the default Thread#getContextClassLoader will be used.
The task is running in a persist shared [thread pool](https://github.com/apache/iceberg/blob/5009949ba4377ac5a8572ff7ae70e886c9e33bec/core/src/main/java/org/apache/iceberg/util/ThreadPools.java?#L60-L62), whose context class loader will be closed in Flink TaskManager, after being used for the first time.
However just setting the class loader manually is not enough, because others code may still use the context class loader, which can not be avoided.
An example exception stack like following, which is caused by ServiceLoader#load in JDK,
<details>
<summary>at javax.xml.parsers.FactoryFinder.findServiceProvider</summary>
```
java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs
in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'.
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:159)
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResources(FlinkUserCodeClassLoaders.java:188)
at java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1196)
at java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221)
at java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
at java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
at java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
at javax.xml.parsers.FactoryFinder$1.run(FactoryFinder.java:287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.xml.parsers.FactoryFinder.findServiceProvider(FactoryFinder.java:283)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:261)
at javax.xml.parsers.SAXParserFactory.newInstance(SAXParserFactory.java:147)
at org.jdom.input.JAXPParserFactory.createParser(JAXPParserFactory.java:125)
at jdk.internal.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:566)
at org.jdom.input.SAXBuilder.createParser(SAXBuilder.java:585)
at org.jdom.input.SAXBuilder.build(SAXBuilder.java:460)
at org.jdom.input.SAXBuilder.build(SAXBuilder.java:807)
at com.aliyun.oss.internal.ResponseParsers.getXmlRootElement(ResponseParsers.java:1015)
at com.aliyun.oss.internal.ResponseParsers.parseListObjects(ResponseParsers.java:1028)
at com.aliyun.oss.internal.ResponseParsers$ListObjectsReponseParser.parse(ResponseParsers.java:562)
at com.aliyun.oss.internal.ResponseParsers$ListObjectsReponseParser.parse(ResponseParsers.java:556)
at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:152)
at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113)
at com.aliyun.oss.internal.OSSBucketOperation.listObjects(OSSBucketOperation.java:421)
at com.aliyun.oss.OSSClient.listObjects(OSSClient.java:445)
at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.listObjects(AliyunOSSFileSystemStore.java:434)
at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.getFileStatus(AliyunOSSFileSystem.java:273)
at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.create(AliyunOSSFileSystem.java:115)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987)
at org.apache.iceberg.hadoop.HadoopOutputFile.createOrOverwrite(HadoopOutputFile.java:85)
at org.apache.iceberg.avro.AvroFileAppender.<init>(AvroFileAppender.java:51)
at org.apache.iceberg.avro.Avro$WriteBuilder.build(Avro.java:198)
at org.apache.iceberg.ManifestWriter$V1Writer.newAppender(ManifestWriter.java:277)
at org.apache.iceberg.ManifestWriter.<init>(ManifestWriter.java:58)
at org.apache.iceberg.ManifestWriter.<init>(ManifestWriter.java:34)
at org.apache.iceberg.ManifestWriter$V1Writer.<init>(ManifestWriter.java:256)
at org.apache.iceberg.ManifestFiles.write(ManifestFiles.java:117)
at org.apache.iceberg.SnapshotProducer.newManifestWriter(SnapshotProducer.java:370)
at org.apache.iceberg.MergingSnapshotProducer$DataFileFilterManager.newManifestWriter(MergingSnapshotProducer.java:711)
at org.apache.iceberg.ManifestFilterManager.filterManifestWithDeletedFiles(ManifestFilterManager.java:383)
at org.apache.iceberg.ManifestFilterManager.filterManifest(ManifestFilterManager.java:308)
at org.apache.iceberg.ManifestFilterManager.lambda$filterManifests$0(ManifestFilterManager.java:186)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:405)
at org.apache.iceberg.util.Tasks$Builder.access$300(Tasks.java:71)
at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:311)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:829)
```
</details>
So looks like we should avoid using shared thread pool across different job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org