You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Wei Zhong (Jira)" <ji...@apache.org> on 2020/04/27 12:41:00 UTC
[jira] [Updated] (FLINK-17417) The flink-avro-sql-jar is not really
sql-jar
[ https://issues.apache.org/jira/browse/FLINK-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei Zhong updated FLINK-17417:
------------------------------
Description:
The flink-avro-\{flink.version}-sql-jar.jar is not really sql-jar. It does not contains the transitive dependence of flink-avro, i.e. Apache Avro. It can be reproduced by the following code:
{code:java}
URL flinkAvroSqlJar = new URL("file:///Users/zhongwei/flink/flink-formats/flink-avro/target/flink-avro-1.11-SNAPSHOT-sql-jar.jar");
URLClassLoader classLoader = new URLClassLoader(new URL[]{flinkAvroSqlJar}, Thread.currentThread().getContextClassLoader());
Thread.currentThread().setContextClassLoader(classLoader);
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
List<Tuple2> list = new ArrayList<>();
list.add(new Tuple2(1, 2));
list.add(new Tuple2(3, 4));
DataStreamUtils.collect(env.fromCollection(list, Types.GENERIC(Tuple2.class)));
{code}
The exception is:
{code:java}
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/generic/GenericData$Array
at org.apache.flink.formats.avro.utils.AvroKryoSerializerUtils.addAvroGenericDataArrayRegistration(AvroKryoSerializerUtils.java:69)
at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.buildKryoRegistrations(KryoSerializer.java:565)
at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.<init>(KryoSerializer.java:132)
at org.apache.flink.api.java.typeutils.GenericTypeInfo.createSerializer(GenericTypeInfo.java:90)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromCollection(StreamExecutionEnvironment.java:814)
at com.example.TestArgs.main(TestArgs.java:24)
Caused by: java.lang.ClassNotFoundException: org.apache.avro.generic.GenericData$Array
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 6 more
{code}
was:
The flink-avro-{flink.version}-sql-jar.jar is not really sql-jar. It does not contains the transitive dependence of flink-avro, i.e. Apache Avro. It can be reproduced by the following code:
{code:java}
URL flinkAvroSqlJar = new URL("file:///Users/zhongwei/flink/flink-formats/flink-avro/target/flink-avro-1.11-SNAPSHOT-sql-jar.jar");
URLClassLoader classLoader = new URLClassLoader(new URL[]{flinkAvroSqlJar}, Thread.currentThread().getContextClassLoader());
Thread.currentThread().setContextClassLoader(classLoader);
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
List<Tuple2> list = new ArrayList<>();
list.add(new Tuple2(1, 2));
list.add(new Tuple2(3, 4));
DataStreamUtils.collect(env.fromCollection(list, Types.GENERIC(Tuple2.class)));
{code}
The exception is:
{code:java}
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/generic/GenericData$Array
at org.apache.flink.formats.avro.utils.AvroKryoSerializerUtils.addAvroGenericDataArrayRegistration(AvroKryoSerializerUtils.java:69)
at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.buildKryoRegistrations(KryoSerializer.java:565)
at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.<init>(KryoSerializer.java:132)
at org.apache.flink.api.java.typeutils.GenericTypeInfo.createSerializer(GenericTypeInfo.java:90)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromCollection(StreamExecutionEnvironment.java:814)
at com.example.TestArgs.main(TestArgs.java:24)
Caused by: java.lang.ClassNotFoundException: org.apache.avro.generic.GenericData$Array
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 6 more
{code}
> The flink-avro-sql-jar is not really sql-jar
> --------------------------------------------
>
> Key: FLINK-17417
> URL: https://issues.apache.org/jira/browse/FLINK-17417
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Affects Versions: 1.9.3, 1.10.0, 1.11.0
> Reporter: Wei Zhong
> Priority: Major
>
> The flink-avro-\{flink.version}-sql-jar.jar is not really sql-jar. It does not contains the transitive dependence of flink-avro, i.e. Apache Avro. It can be reproduced by the following code:
> {code:java}
> URL flinkAvroSqlJar = new URL("file:///Users/zhongwei/flink/flink-formats/flink-avro/target/flink-avro-1.11-SNAPSHOT-sql-jar.jar");
> URLClassLoader classLoader = new URLClassLoader(new URL[]{flinkAvroSqlJar}, Thread.currentThread().getContextClassLoader());
> Thread.currentThread().setContextClassLoader(classLoader);
> StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
> List<Tuple2> list = new ArrayList<>();
> list.add(new Tuple2(1, 2));
> list.add(new Tuple2(3, 4));
> DataStreamUtils.collect(env.fromCollection(list, Types.GENERIC(Tuple2.class)));
> {code}
> The exception is:
> {code:java}
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/generic/GenericData$Array
> at org.apache.flink.formats.avro.utils.AvroKryoSerializerUtils.addAvroGenericDataArrayRegistration(AvroKryoSerializerUtils.java:69)
> at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.buildKryoRegistrations(KryoSerializer.java:565)
> at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.<init>(KryoSerializer.java:132)
> at org.apache.flink.api.java.typeutils.GenericTypeInfo.createSerializer(GenericTypeInfo.java:90)
> at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.fromCollection(StreamExecutionEnvironment.java:814)
> at com.example.TestArgs.main(TestArgs.java:24)
> Caused by: java.lang.ClassNotFoundException: org.apache.avro.generic.GenericData$Array
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 6 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)