Issues with Samza-Kafka compatibility while upgrading to latest versions

["Choudhary, Suraj" <Su...@pearson.com>]

Hi,

I am working on updating our infrastructure by upgrading Samza, Kafka and Hadoop(our container manager for samza).

I am upgrading from/to these versions:
Samza:    from 2.11-0.12.0 to 2.12-1.4.0
Kafka:      from 2.11-0.10.0.1 to 2.12-2.5.0
Hadoop:  from 2.7.2 to 2.9.2

After updating the versions on starting the Samza application I am getting the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: kafka/common/TopicAndPartition
at org.apache.samza.system.kafka.KafkaSystemConsumer.toTopicAndPartition(KafkaSystemConsumer.java:317)
at org.apache.samza.system.kafka.KafkaSystemConsumer.register(KafkaSystemConsumer.java:284)
at org.apache.samza.coordinator.stream.CoordinatorStreamSystemConsumer.register(CoordinatorStreamSystemConsumer.java:130)
at org.apache.samza.util.CoordinatorStreamUtil$.writeConfigToCoordinatorStream(CoordinatorStreamUtil.scala:159)
at org.apache.samza.job.JobRunner.run(JobRunner.scala:80)
at org.apache.samza.job.JobRunner$.doOperation(JobRunner.scala:52)
at org.apache.samza.job.JobRunner$.main(JobRunner.scala:47)
at org.apache.samza.job.JobRunner.main(JobRunner.scala)
Caused by: java.lang.ClassNotFoundException: kafka.common.TopicAndPartition
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 8 more

On further checking I found that ‘KafkaSystemConsumer’ is part of samza-kafka_2.12-1.4.0.jar. And I looked into our deploy jars and I am finding only one copy of this class.
The reference class ‘TopicAndPartition’ is not present anywhere in the deploy jars or kafka_2.12-2.5.0 jar in particular.

I tried to find a Samza-Kafka version compatibility documentation. But I didn’t find any. On investigating our import of samza-kafka_2.12-1.4.0.jar, I found that it is including the kafka_2.12-2.5.0 dependency, which leads me to think that Samza should be compatible the Kafka version I am referencing.

I am also including all the required maven dependencies such as kafka, kafka-clients, samza-core, samza-api of these latest versions and made sure no other version of the same is present in my deploy jars.

Please let me know If I am missing something. And what can do to resolve this issue.

Thanks in Advance,
Suraj

Re: Issues with Samza-Kafka compatibility while upgrading to latest versions

[Yi Pan <ni...@gmail.com>]

Hi, Choudhary,

Thanks for reporting your issues. Samza manages its dependencies in a few
gradle files under ${ROOT}/gradle/dependency-versions-*.gradle.
Specifically, the Kafka dependency is defined in the file
${ROOT}/gradle/dependency-versions.gradle. I quickly checked the versions
listed in 1.4.0 and it is still at 2.0.1. Please try to use 2.0.1 instead
of 2.5.0. Let me know if that works for you.

Thanks!

-Yi

On Thu, Oct 8, 2020 at 1:48 PM Choudhary, Suraj <Su...@pearson.com>
wrote:

> Hi,
>
> I am working on updating our infrastructure by upgrading Samza, Kafka and
> Hadoop(our container manager for samza).
>
> I am upgrading from/to these versions:
> Samza:    from 2.11-0.12.0 to 2.12-1.4.0
> Kafka:      from 2.11-0.10.0.1 to 2.12-2.5.0
> Hadoop:  from 2.7.2 to 2.9.2
>
> After updating the versions on starting the Samza application I am getting
> the following error:
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> kafka/common/TopicAndPartition
> at
> org.apache.samza.system.kafka.KafkaSystemConsumer.toTopicAndPartition(KafkaSystemConsumer.java:317)
> at
> org.apache.samza.system.kafka.KafkaSystemConsumer.register(KafkaSystemConsumer.java:284)
> at
> org.apache.samza.coordinator.stream.CoordinatorStreamSystemConsumer.register(CoordinatorStreamSystemConsumer.java:130)
> at
> org.apache.samza.util.CoordinatorStreamUtil$.writeConfigToCoordinatorStream(CoordinatorStreamUtil.scala:159)
> at org.apache.samza.job.JobRunner.run(JobRunner.scala:80)
> at org.apache.samza.job.JobRunner$.doOperation(JobRunner.scala:52)
> at org.apache.samza.job.JobRunner$.main(JobRunner.scala:47)
> at org.apache.samza.job.JobRunner.main(JobRunner.scala)
> Caused by: java.lang.ClassNotFoundException: kafka.common.TopicAndPartition
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 8 more
>
> On further checking I found that ‘KafkaSystemConsumer’ is part of
> samza-kafka_2.12-1.4.0.jar. And I looked into our deploy jars and I am
> finding only one copy of this class.
> The reference class ‘TopicAndPartition’ is not present anywhere in the
> deploy jars or kafka_2.12-2.5.0 jar in particular.
>
> I tried to find a Samza-Kafka version compatibility documentation. But I
> didn’t find any. On investigating our import of samza-kafka_2.12-1.4.0.jar,
> I found that it is including the kafka_2.12-2.5.0 dependency, which leads
> me to think that Samza should be compatible the Kafka version I am
> referencing.
>
> I am also including all the required maven dependencies such as kafka,
> kafka-clients, samza-core, samza-api of these latest versions and made sure
> no other version of the same is present in my deploy jars.
>
> Please let me know If I am missing something. And what can do to resolve
> this issue.
>
> Thanks in Advance,
> Suraj
>
>
>