You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/07/06 15:45:33 UTC
[GitHub] [flink] DavidFricker commented on pull request #15491: [FLINK-21966][python]Support Kinesis connector in Python DataStream API
DavidFricker commented on pull request #15491:
URL: https://github.com/apache/flink/pull/15491#issuecomment-874873342
We have implemented something similar and started receiving classloader exceptions as below, however the job continues to work fine and stream data from Kinesis:
```
Job has been submitted with JobID 31975670b3660829abf7e69c3a13c2c6
Exception in thread "Thread-11" java.lang.IllegalStateException: Trying to access closed classloader. Please check if you store classloaders directly or indirectly in static fields. If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately, you can disable this check with the configuration 'classloader.check-leaked-classloader'.
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:164)
at org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResource(FlinkUserCodeClassLoaders.java:183)
at java.lang.ClassLoader.getResource(ClassLoader.java:1084)
at org.apache.hadoop.conf.Configuration.getResource(Configuration.java:2779)
at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3035)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2994)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2967)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2847)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1199)
at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1811)
at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1788)
at org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:183)
at org.apache.hadoop.util.ShutdownHookManager.shutdownExecutor(ShutdownHookManager.java:145)
at org.apache.hadoop.util.ShutdownHookManager.access$300(ShutdownHookManager.java:65)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:102)
```
We also tried the implementation from this PR and got the same error as above.
Happy to contribute back here and help to debug if someone can point us in the right direction. We are using:
- Flink 1.13.1
- apache-flink==1.13.1 (for pyflink)
- YARN on AWS EMR
- `flink-sql-connector-kinesis_2.11-1.13.1.jar`
- `flink run -t yarn-per-job --detached -py /path/to/main.py --pyFiles /path/to/dependencies.zip -pyexec /usr/bin/python3.7 --parallelism 4` to submit the job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org