You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Chesnay Schepler (Jira)" <ji...@apache.org> on 2020/05/28 07:33:00 UTC
[jira] [Closed] (FLINK-17989) java.lang.NoClassDefFoundError:
org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter
[ https://issues.apache.org/jira/browse/FLINK-17989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chesnay Schepler closed FLINK-17989.
------------------------------------
Fix Version/s: (was: 1.11.1)
(was: 1.10.1)
(was: 1.9.3)
(was: 1.11.0)
Resolution: Duplicate
> java.lang.NoClassDefFoundError: org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter
> -----------------------------------------------------------------------------------------------
>
> Key: FLINK-17989
> URL: https://issues.apache.org/jira/browse/FLINK-17989
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, FileSystems
> Affects Versions: 1.9.3, 1.10.1, 1.11.0, 1.11.1
> Environment: Ubuntu 18
> Java 1.8
> Flink 1.9.x and 1.10.x
> Reporter: Israel Ekpo
> Priority: Critical
>
> In the POM.xml classes from certain packages are relocated and filtered out of the final jar In the POM.xml classes from certain packages are relocated and filtered out of the final jar
> This is causing errors for customers and users that are using the StreamingFileSink with Azure Blob Storage in Flink versions 1.9.x, 1.10.x and possibly 1.11.x
> [https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170|https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233]
> [https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233|https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-azure-fs-hadoop/pom.xml#L170https://github.com/apache/flink/blob/release-1.9/flink-filesystems/flink-fs-hadoop-shaded/pom.xml#L233]
>
> I would like to know why the relocation is happening and the reasoning behind the exclusion and filtering of the classes
> It seems to affect just the Azure file systems in my sample implementations
>
> {code:java}
> String outputPath = "wasbs://container@account.blob.core.windows.net/streaming-output/";
> final StreamingFileSink<Integer> sink = StreamingFileSink .forRowFormat(new Path(outputPath), new SimpleStringEncoder<Integer>("UTF-8")) .build();
> stream.addSink(sink);
> // execute programenv.execute(StreamingJob.class.getCanonicalName());
> {code}
> {code:java}
> //Exception Details Below
> 2020-05-27 17:23:16java.lang.NoClassDefFoundError: org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter at org.apache.flink.fs.azure.common.hadoop.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202) at org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69) at org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:112) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:242) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:327) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281) at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:881) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:395) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.ClassNotFoundException: org.apache.flink.fs.azure.common.hadoop.HadoopRecoverableWriter at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at org.apache.flink.util.ChildFirstClassLoader.loadClass(ChildFirstClassLoader.java:60) at java.lang.ClassLoader.loadClass(ClassLoader.java:351){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)