You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@storm.apache.org by "Raja.Aravapalli" <Ra...@target.com> on 2016/09/17 03:46:21 UTC

storm: kafka to hdfs - dependency jars issues

Hi,

I have built a storm topology, that consumes data from kafka and writes into hdfs.


When using storm & kafka dependencies like below, topology is failing with below stated exception:


<storm.version>0.10.0.2.3.4.0-3485</storm.version>

<kafka.version>0.8.2.1</kafka.version>


java.lang.NoSuchMethodError: kafka.javaapi.consumer.SimpleConsumer.<init>(Ljava/lang/String;IIILjava/lang/String;Ljava/lang/String;)V at storm.kafka.DynamicPartitionConnections.register(DynamicPartitionConnections.java:60) at storm.kafka.PartitionManager.<init>(PartitionManager.java:66) at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:138) at backtype.storm.daemon.executor$fn__7098$fn__7113$fn__7142.invoke(executor.clj:596) at backtype.storm.util$async_loop$fn__543.invoke(util.clj:475) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)



But, When using storm & kafka dependencies like below topology is running fine and downloading messages and writing to hdfs , without any issues!!


<storm.version>0.9.3.2.2.4.0-2633</storm.version>

<kafka.version>0.8.2.1</kafka.version>


Can someone please help what is the causing the issue and help me fix the issue !


Regards,
Raja.

Re: storm: kafka to hdfs - dependency jars issues

Posted by Jagat Singh <ja...@gmail.com>.

Hello Raja,

Seems you are using Hortonworks cluster.

You should choose the required binary versions from Hortonworks maven repo

http://repo.hortonworks.com/content/repositories/releases/org/apache/

Example the HDP version 2.3.4.0-3485 should match in all binaries you use.

http://repo.hortonworks.com/content/repositories/releases/org/apache/storm/storm-kafka/0.10.0.2.3.4.0-3485/

Thanks,

Jagat Singh







On 17 September 2016 at 13:46, Raja.Aravapalli <Ra...@target.com>
wrote:

>
>
> Hi,
>
>
>
> I have built a storm topology, that consumes data from kafka and writes
> into hdfs.
>
>
>
>
>
> When using storm & kafka dependencies like below, topology is failing with
> below stated exception:
>
>
>
> <*storm.version*>0.10.0.2.3.4.0-3485</*storm.version*>
>
> <*kafka.version*>0.8.2.1</*kafka.version*>
>
>
>
>
>
> java.lang.NoSuchMethodError: kafka.javaapi.consumer.
> SimpleConsumer.<init>(Ljava/lang/String;IIILjava/lang/String;Ljava/lang/String;)V
> at storm.kafka.DynamicPartitionConnections.register(
> DynamicPartitionConnections.java:60) at storm.kafka.PartitionManager.<
> init>(PartitionManager.java:66) at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98)
> at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69)
> at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:138) at
> backtype.storm.daemon.executor$fn__7098$fn__7113$fn__7142.invoke(executor.clj:596)
> at backtype.storm.util$async_loop$fn__543.invoke(util.clj:475) at
> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
>
>
> But, When using storm & kafka dependencies like below topology is running
> fine and downloading messages and writing to hdfs , without any issues!!
>
>
>
> <*storm.version*>0.9.3.2.2.4.0-2633</*storm.version*>
>
> <*kafka.version*>0.8.2.1</*kafka.version*>
>
>
>
>
>
> Can someone please help what is the causing the issue and help me fix the
> issue !
>
>
>
>
>
> Regards,
>
> Raja.
>