You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Israel Ekpo (Jira)" <ji...@apache.org> on 2020/05/27 21:45:00 UTC
[jira] [Created] (FLINK-17989) java.lang.NoClassDefFoundError:
org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter
Israel Ekpo created FLINK-17989:
-----------------------------------
Summary: java.lang.NoClassDefFoundError: org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter
Key: FLINK-17989
URL: https://issues.apache.org/jira/browse/FLINK-17989
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem, FileSystems
Affects Versions: 1.10.1, 1.9.3
Environment: Ubuntu 18
Java 1.8
Flink 1.9.x and 1.10.x
Reporter: Israel Ekpo
Fix For: 1.10.1, 1.9.3
In the POM.xml classes from certain packages are relocated and filtered out of the final jar In the POM.xml classes from certain packages are relocated and filtered out of the final jar
This is causing errors for customers and users that are using the StreamingFileSink with Azure Blob Storage in Flink version 1.9.x, 1.10.x and possibly 1.11.x
https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233
I would like to know why the relocation is happening and the reasoning behind the exclusion and filtering of the classes
It seems to affect just the Azure file systems in my sample implementations
{code:java}
String outputPath = "wasbs://container@account.blob.core.windows.net/streaming-output/";
final StreamingFileSink<Integer> sink = StreamingFileSink .forRowFormat(new Path(outputPath), new SimpleStringEncoder<Integer>("UTF-8")) .build();
stream.addSink(sink);
// execute programenv.execute(StreamingJob.class.getCanonicalName());
{code}
2020-05-27 17:23:16java.lang.NoClassDefFoundError: org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter at org.apache.flink.fs.azure.common.hadoop.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202) at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69) at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:112) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:242) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281) at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:881) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:395) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.ClassNotFoundException: org.apache.flink.fs.azure.common.hadoop.HadoopRecoverableWriter at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60) at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)