You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/12/20 04:21:41 UTC
[GitHub] [iceberg] yittg opened a new issue #3776: Flink connector thrown `Trying to access closed classloader`
yittg opened a new issue #3776:
URL: https://github.com/apache/iceberg/issues/3776
Hi, i'm trying to test the unreleased Flink 1.13 connector on a Flink 1.13.1 cluster running on k8s.
The task is simple like `INSERT OVERWRITE a SELECT * FROM b`.
When i submit a task for the first time, it's ok.
But when i submit the task for the second time(run on the same TaskManager with the first time), the following exception is thrown, it's weird.
<details>
<summary>java.lang.IllegalStateException: Trying to access closed classloader</summary>
```
2021-12-20 11:53:11,697 INFO org.apache.flink.runtime.taskmanager.Task [] - IcebergFilesCommitter -> Sink: IcebergSink ice.t1.d (1/1)#0 (e74babbac1ebb51fe72958bd177ae534) switched from INITIALIZING to RUNNING.
2021-12-20 11:53:11,854 INFO org.apache.iceberg.flink.sink.IcebergFilesCommitter [] - Committing dynamic partition overwrite with 1 data files and 0 delete files to table ice.t1.d
2021-12-20 11:53:11,946 WARN org.apache.flink.runtime.taskmanager.Task [] - IcebergFilesCommitter -> Sink: IcebergSink ice.t1.d (1/1)#0 (e74babbac1ebb51fe72958bd177ae534) switched from RUNNING to FAILED with failure cause: java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'.
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:164)
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:172)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Unknown Source)
at org.apache.iceberg.common.DynClasses$Builder.impl(DynClasses.java:68)
at org.apache.iceberg.avro.GenericAvroReader$ReadBuilder.record(GenericAvroReader.java:85)
at org.apache.iceberg.avro.GenericAvroReader$ReadBuilder.record(GenericAvroReader.java:72)
at org.apache.iceberg.avro.AvroSchemaVisitor.visit(AvroSchemaVisitor.java:50)
at org.apache.iceberg.avro.GenericAvroReader.initReader(GenericAvroReader.java:47)
at org.apache.iceberg.avro.GenericAvroReader.setSchema(GenericAvroReader.java:53)
at org.apache.iceberg.avro.ProjectionDatumReader.newDatumReader(ProjectionDatumReader.java:80)
at org.apache.iceberg.avro.ProjectionDatumReader.setSchema(ProjectionDatumReader.java:69)
at org.apache.iceberg.shaded.org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:133)
at org.apache.iceberg.shaded.org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:130)
at org.apache.iceberg.shaded.org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:122)
at org.apache.iceberg.shaded.org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:66)
at org.apache.iceberg.avro.AvroIterable.newFileReader(AvroIterable.java:100)
at org.apache.iceberg.avro.AvroIterable.getMetadata(AvroIterable.java:66)
at org.apache.iceberg.ManifestReader.<init>(ManifestReader.java:103)
at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:87)
at org.apache.iceberg.SnapshotProducer.newManifestReader(SnapshotProducer.java:378)
at org.apache.iceberg.MergingSnapshotProducer$DataFileFilterManager.newManifestReader(MergingSnapshotProducer.java:716)
at org.apache.iceberg.ManifestFilterManager.filterManifest(ManifestFilterManager.java:295)
at org.apache.iceberg.ManifestFilterManager.lambda$filterManifests$0(ManifestFilterManager.java:186)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:405)
at org.apache.iceberg.util.Tasks$Builder.access$300(Tasks.java:71)
at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:311)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue closed issue #3776: Flink connector thrown `Trying to access closed classloader`
Posted by GitBox <gi...@apache.org>.
rdblue closed issue #3776:
URL: https://github.com/apache/iceberg/issues/3776
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yittg commented on issue #3776: Flink connector thrown `Trying to access closed classloader`
Posted by GitBox <gi...@apache.org>.
yittg commented on issue #3776:
URL: https://github.com/apache/iceberg/issues/3776#issuecomment-999427065
To be clear, i encountered this issue by putting the iceberg Flink runtime jar into the lib directory as [Flink suggested](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code).
So i tested another scenario next, submit the iceberg jar as user code.
As expected, it can work well for the first several jobs, then the followed job will fail on meta space OOM
```
java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
```
You can also see many `iceberg-worker-pool-0` threads because the worker pool can not be released along with the loaded user classes.
So either we need pass a ExecutorService to the tasks from the Flink context, or provide a mechanism to release these threads after job complete, like making use of `RuntimeContext#registerUserCodeClassLoaderReleaseHookIfAbsent`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yittg commented on issue #3776: Flink connector thrown `Trying to access closed classloader`
Posted by GitBox <gi...@apache.org>.
yittg commented on issue #3776:
URL: https://github.com/apache/iceberg/issues/3776#issuecomment-1000117806
How about fixes this issue simply as #3797 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yittg edited a comment on issue #3776: Flink connector thrown `Trying to access closed classloader`
Posted by GitBox <gi...@apache.org>.
yittg edited a comment on issue #3776:
URL: https://github.com/apache/iceberg/issues/3776#issuecomment-999427065
To be clear, i encountered this issue by putting the iceberg Flink runtime jar into the lib directory as [Flink suggested](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code).
So i tested another scenario next, submit the iceberg jar as user code.
As expected, it can work well for the first several jobs, then the followed job will fail on meta space OOM
```
java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
```
You can also see many `iceberg-worker-pool-0` threads because the worker pool can not be released along with the loaded user classes.
So either we need pass a ExecutorService to the tasks from the Flink context, ~or provide a mechanism to release these threads after job complete, like making use of `RuntimeContext#registerUserCodeClassLoaderReleaseHookIfAbsent`.~
> release a shared pool may be not a good choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yittg commented on issue #3776: Flink connector thrown `Trying to access closed classloader`
Posted by GitBox <gi...@apache.org>.
yittg commented on issue #3776:
URL: https://github.com/apache/iceberg/issues/3776#issuecomment-1009562077
@openinx Would you help to check this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #3776: Flink connector thrown `Trying to access closed classloader`
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #3776:
URL: https://github.com/apache/iceberg/issues/3776#issuecomment-1015946568
I merged #3906. Ping me on the next one and I'll review that as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yittg edited a comment on issue #3776: Flink connector thrown `Trying to access closed classloader`
Posted by GitBox <gi...@apache.org>.
yittg edited a comment on issue #3776:
URL: https://github.com/apache/iceberg/issues/3776#issuecomment-999427065
To be clear, i encountered this issue by putting the iceberg Flink runtime jar into the lib directory as [Flink suggested](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code).
So i tested another scenario next, submit the iceberg jar as user code.
As expected, it can work well for the first several jobs, then the followed job will fail on meta space OOM
```
java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
```
You can also see many `iceberg-worker-pool-0` threads because the worker pool can not be released along with the loaded user classes.
So either we need pass a ExecutorService to the tasks from the Flink context, ~or provide a mechanism to release these threads after job complete, like making use of `RuntimeContext#registerUserCodeClassLoaderReleaseHookIfAbsent`.~
> release shared pool may be not a good choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yittg commented on issue #3776: Flink connector thrown `Trying to access closed classloader`
Posted by GitBox <gi...@apache.org>.
yittg commented on issue #3776:
URL: https://github.com/apache/iceberg/issues/3776#issuecomment-1000599936
@rdblue Would you help to check this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwangben commented on issue #3776: Flink connector thrown `Trying to access closed classloader`
Posted by GitBox <gi...@apache.org>.
wuwangben commented on issue #3776:
URL: https://github.com/apache/iceberg/issues/3776#issuecomment-1010759584
MetaSpace OOM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yittg edited a comment on issue #3776: Flink connector thrown `Trying to access closed classloader`
Posted by GitBox <gi...@apache.org>.
yittg edited a comment on issue #3776:
URL: https://github.com/apache/iceberg/issues/3776#issuecomment-998502908
After some digging, i find the direct cause is [that](https://github.com/apache/iceberg/blob/5009949ba4377ac5a8572ff7ae70e886c9e33bec/core/src/main/java/org/apache/iceberg/ManifestReader.java?#L100-L102):
when we build a `AvroIterable`, we didn't set the class loader, then the default Thread#getContextClassLoader will be used.
The task is running in a persist shared [thread pool](https://github.com/apache/iceberg/blob/5009949ba4377ac5a8572ff7ae70e886c9e33bec/core/src/main/java/org/apache/iceberg/util/ThreadPools.java?#L60-L62), whose context class loader will be closed in Flink TaskManager, after being used for the first time.
However just setting the class loader manually is not enough, because others code may still use the context class loader, which can not be avoided.
An example exception stack like following, which is caused by ServiceLoader#load in JDK,
<details>
<summary>at javax.xml.parsers.FactoryFinder.findServiceProvider</summary>
```
java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs
in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'.
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:159)
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResources(FlinkUserCodeClassLoaders.java:188)
at java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1196)
at java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221)
at java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
at java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
at java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
at javax.xml.parsers.FactoryFinder$1.run(FactoryFinder.java:287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.xml.parsers.FactoryFinder.findServiceProvider(FactoryFinder.java:283)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:261)
at javax.xml.parsers.SAXParserFactory.newInstance(SAXParserFactory.java:147)
at org.jdom.input.JAXPParserFactory.createParser(JAXPParserFactory.java:125)
at jdk.internal.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:566)
at org.jdom.input.SAXBuilder.createParser(SAXBuilder.java:585)
at org.jdom.input.SAXBuilder.build(SAXBuilder.java:460)
at org.jdom.input.SAXBuilder.build(SAXBuilder.java:807)
at com.aliyun.oss.internal.ResponseParsers.getXmlRootElement(ResponseParsers.java:1015)
at com.aliyun.oss.internal.ResponseParsers.parseListObjects(ResponseParsers.java:1028)
at com.aliyun.oss.internal.ResponseParsers$ListObjectsReponseParser.parse(ResponseParsers.java:562)
at com.aliyun.oss.internal.ResponseParsers$ListObjectsReponseParser.parse(ResponseParsers.java:556)
at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:152)
at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113)
at com.aliyun.oss.internal.OSSBucketOperation.listObjects(OSSBucketOperation.java:421)
at com.aliyun.oss.OSSClient.listObjects(OSSClient.java:445)
at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.listObjects(AliyunOSSFileSystemStore.java:434)
at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.getFileStatus(AliyunOSSFileSystem.java:273)
at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.create(AliyunOSSFileSystem.java:115)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987)
at org.apache.iceberg.hadoop.HadoopOutputFile.createOrOverwrite(HadoopOutputFile.java:85)
at org.apache.iceberg.avro.AvroFileAppender.<init>(AvroFileAppender.java:51)
at org.apache.iceberg.avro.Avro$WriteBuilder.build(Avro.java:198)
at org.apache.iceberg.ManifestWriter$V1Writer.newAppender(ManifestWriter.java:277)
at org.apache.iceberg.ManifestWriter.<init>(ManifestWriter.java:58)
at org.apache.iceberg.ManifestWriter.<init>(ManifestWriter.java:34)
at org.apache.iceberg.ManifestWriter$V1Writer.<init>(ManifestWriter.java:256)
at org.apache.iceberg.ManifestFiles.write(ManifestFiles.java:117)
at org.apache.iceberg.SnapshotProducer.newManifestWriter(SnapshotProducer.java:370)
at org.apache.iceberg.MergingSnapshotProducer$DataFileFilterManager.newManifestWriter(MergingSnapshotProducer.java:711)
at org.apache.iceberg.ManifestFilterManager.filterManifestWithDeletedFiles(ManifestFilterManager.java:383)
at org.apache.iceberg.ManifestFilterManager.filterManifest(ManifestFilterManager.java:308)
at org.apache.iceberg.ManifestFilterManager.lambda$filterManifests$0(ManifestFilterManager.java:186)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:405)
at org.apache.iceberg.util.Tasks$Builder.access$300(Tasks.java:71)
at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:311)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:829)
```
</details>
So looks like we should avoid using shared thread pool across different job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yittg commented on issue #3776: Flink connector thrown `Trying to access closed classloader`
Posted by GitBox <gi...@apache.org>.
yittg commented on issue #3776:
URL: https://github.com/apache/iceberg/issues/3776#issuecomment-998502908
After some digging, i find the direct cause is [that](https://github.com/apache/iceberg/blob/5009949ba4377ac5a8572ff7ae70e886c9e33bec/core/src/main/java/org/apache/iceberg/ManifestReader.java?#L100-L102):
when we build a `AvroIterable`, we didn't set the class loader, then the default Thread#getContextClassLoader will be used.
The task is running in a persist shared [thread pool](https://github.com/apache/iceberg/blob/5009949ba4377ac5a8572ff7ae70e886c9e33bec/core/src/main/java/org/apache/iceberg/util/ThreadPools.java?#L60-L62), whose context class loader will be closed in Flink TaskManager, after being used for the first time.
However just setting the class loader manually is not enough, because others code may still use the context class loader, which can not be avoided.
An example exception stack like following, which is caused by ServiceLoader#load in JDK,
<details>
<summary>at javax.xml.parsers.FactoryFinder.findServiceProvider</summary>
```
java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs
in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'.
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:159) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResources(FlinkUserCodeClassLoaders.java:188) ~[flink-dist_2.12-1.13.1.jar:1.13.1]
at java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1196) ~[?:?]
at java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221) ~[?:?]
at java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265) ~[?:?]
at java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300) ~[?:?]
at java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385) ~[?:?]
at javax.xml.parsers.FactoryFinder$1.run(FactoryFinder.java:287) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at javax.xml.parsers.FactoryFinder.findServiceProvider(FactoryFinder.java:283) ~[?:?]
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:261) ~[?:?]
at javax.xml.parsers.SAXParserFactory.newInstance(SAXParserFactory.java:147) ~[?:?]
at org.jdom.input.JAXPParserFactory.createParser(JAXPParserFactory.java:125) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at jdk.internal.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.jdom.input.SAXBuilder.createParser(SAXBuilder.java:585) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.jdom.input.SAXBuilder.build(SAXBuilder.java:460) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.jdom.input.SAXBuilder.build(SAXBuilder.java:807) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at com.aliyun.oss.internal.ResponseParsers.getXmlRootElement(ResponseParsers.java:1015) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at com.aliyun.oss.internal.ResponseParsers.parseListObjects(ResponseParsers.java:1028) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at com.aliyun.oss.internal.ResponseParsers$ListObjectsReponseParser.parse(ResponseParsers.java:562) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at com.aliyun.oss.internal.ResponseParsers$ListObjectsReponseParser.parse(ResponseParsers.java:556) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:152) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at com.aliyun.oss.internal.OSSBucketOperation.listObjects(OSSBucketOperation.java:421) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at com.aliyun.oss.OSSClient.listObjects(OSSClient.java:445) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.listObjects(AliyunOSSFileSystemStore.java:434) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.getFileStatus(AliyunOSSFileSystem.java:273) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.create(AliyunOSSFileSystem.java:115) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.hadoop.HadoopOutputFile.createOrOverwrite(HadoopOutputFile.java:85) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.avro.AvroFileAppender.<init>(AvroFileAppender.java:51) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.avro.Avro$WriteBuilder.build(Avro.java:198) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.ManifestWriter$V1Writer.newAppender(ManifestWriter.java:277) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.ManifestWriter.<init>(ManifestWriter.java:58) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.ManifestWriter.<init>(ManifestWriter.java:34) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.ManifestWriter$V1Writer.<init>(ManifestWriter.java:256) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.ManifestFiles.write(ManifestFiles.java:117) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.SnapshotProducer.newManifestWriter(SnapshotProducer.java:370) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.MergingSnapshotProducer$DataFileFilterManager.newManifestWriter(MergingSnapshotProducer.java:711) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.ManifestFilterManager.filterManifestWithDeletedFiles(ManifestFilterManager.java:383) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.ManifestFilterManager.filterManifest(ManifestFilterManager.java:308) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.ManifestFilterManager.lambda$filterManifests$0(ManifestFilterManager.java:186) ~[dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:405) [dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.util.Tasks$Builder.access$300(Tasks.java:71) [dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:311) [dist-1.1-SNAPSHOT.jar:1.1-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
</details>
So looks like we should avoid using shared thread pool across different job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wuwangben commented on issue #3776: Flink connector thrown `Trying to access closed classloader`
Posted by GitBox <gi...@apache.org>.
wuwangben commented on issue #3776:
URL: https://github.com/apache/iceberg/issues/3776#issuecomment-1010758833
I have the same problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org