NoClassDefFoundError if Kafka classes are assemblied in application jar

["L. C. Hsieh" <vi...@apache.org>]

Hi, Flink developers,

Does anyone encounter the following error?

java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArrayDeserializer
        at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.setDeserializer(FlinkKafkaConsumer.java:322)
        at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:223)
        at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:154)
        at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:139)
        at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:108)

If I put kafka-clients jar into Flink's lib/, Flink can find it. But if I assembly it into the application jar, Flink cannot find it. But based on what I read from Flink doc, Flink does "child-first" resolution on classes. Why it cannot find kafka classes if they are in application jar??

I examined the application jar content. It includes these kafka classes actually.

I tested it with K8S session and job clusters on Flink built from current source. Both have the same error. 

Re: NoClassDefFoundError if Kafka classes are assemblied in application jar

["L. C. Hsieh" <vi...@gmail.com>]

Hi Arvid,

Alexander Preuß has already replied to me and I also found a discussion on
https://stackoverflow.com/questions/51479657/flinkkafkaconsumer011-not-found-on-flink-cluster.

So by following
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/dev/datastream/project-configuration/#adding-connector-and-library-dependencies,
that is to remove "provided" dependency scope and let Maven include it
in fat jar, it can work.

So, if we do not put the connector jar under Flink /lib, but include
it in the fat jar of the application. Everything is fine.

I have replied to Alexander a further question as the follows:

But I am also curious about how to explain it? Actually the
information you pointed out said it is recommended to do it, but not
said "cannot put into /lib".
As I mentioned at the beginning, Flink does "child-first" resolution
on classes, why Flink cannot find Kafka classes in the application jar
if the connector jar is under /lib?
It is pretty confusing from a user's point of view.

On Mon, Oct 18, 2021 at 2:01 AM Arvid Heise <ar...@apache.org> wrote:
>
> This looks very odd. How do you create the fat jar? What's your Flink version?
>
> I don't think this is a general Flink issue or else no one would be able to read from Kafka at all.
>
> On Fri, Oct 15, 2021 at 4:16 AM L. C. Hsieh <vi...@apache.org> wrote:
>>
>> Hi, Flink developers,
>>
>> Does anyone encounter the following error?
>>
>> java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/ByteArrayDeserializer
>>         at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.setDeserializer(FlinkKafkaConsumer.java:322)
>>         at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:223)
>>         at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:154)
>>         at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:139)
>>         at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:108)
>>
>> If I put kafka-clients jar into Flink's lib/, Flink can find it. But if I assembly it into the application jar, Flink cannot find it. But based on what I read from Flink doc, Flink does "child-first" resolution on classes. Why it cannot find kafka classes if they are in application jar??
>>
>> I examined the application jar content. It includes these kafka classes actually.
>>
>> I tested it with K8S session and job clusters on Flink built from current source. Both have the same error.

Re: NoClassDefFoundError if Kafka classes are assemblied in application jar

[Arvid Heise <ar...@apache.org>]

This looks very odd. How do you create the fat jar? What's your Flink
version?

I don't think this is a general Flink issue or else no one would be able to
read from Kafka at all.

On Fri, Oct 15, 2021 at 4:16 AM L. C. Hsieh <vi...@apache.org> wrote:

> Hi, Flink developers,
>
> Does anyone encounter the following error?
>
> java.lang.NoClassDefFoundError:
> org/apache/kafka/common/serialization/ByteArrayDeserializer
>         at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.setDeserializer(FlinkKafkaConsumer.java:322)
>         at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:223)
>         at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:154)
>         at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:139)
>         at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:108)
>
> If I put kafka-clients jar into Flink's lib/, Flink can find it. But if I
> assembly it into the application jar, Flink cannot find it. But based on
> what I read from Flink doc, Flink does "child-first" resolution on classes.
> Why it cannot find kafka classes if they are in application jar??
>
> I examined the application jar content. It includes these kafka classes
> actually.
>
> I tested it with K8S session and job clusters on Flink built from current
> source. Both have the same error.
>