You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by pradeep s <sr...@gmail.com> on 2017/02/25 09:27:29 UTC
Storm kafka spout no such method error
Hi,
I am using Storm 1.0.3 and kafka 0.10.1.1 and initialising KafkaSpout using
storm-kafka-client
https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md
Dependencies used are
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.1.1</version>
</dependency>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-kafka-client</artifactId>
<version>1.0.3</version>
</dependency>
While starting the topology spout is failing by below error
java.lang.*NoSuchMethodError*:
org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/kafka/clients/consumer/ConsumerRebalanceListener;)V
at
org.apache.storm.kafka.spout.KafkaSpout.subscribeKafkaConsumer(KafkaSpout.java:360)
~[stormjar.jar:?]
at
org.apache.storm.kafka.spout.KafkaSpout.activate(KafkaSpout.java:351)
~[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__4905$fn__4920$fn__4951.invoke(executor.clj:639)
~[storm-core-1.0.3.jar:1.0.3]
at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
[storm-core-1.0.3.jar:1.0.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Regards
Pradeep S
Re: Storm kafka spout no such method error
Posted by Jungtaek Lim <ka...@gmail.com>.
Unfortunately, storm-kafka-client 1.0.x is compatible with Kafka 0.9 and
storm-kafka-client 1.1.0 (not released yet but we'll make it happen) is
compatible with Kafka 0.10.
https://github.com/apache/storm/blob/1.0.x-branch/external/storm-kafka-client/README.md
https://github.com/apache/storm/blob/1.x-branch/docs/storm-kafka-client.md
You may want to participate release vote for 1.1.0 RC2 and make it happen
soon.
Thanks,
Jungtaek Lim (HeartSaVioR)
2017년 2월 26일 (일) 오전 9:21, Erik Weathers <ew...@groupon.com>님이 작성:
> You must be doing *something* wrong.
>
> One thing that stands out to me is that you're using a different version
> of the kafka-client (0.10.1.1) than storm-kafka-client has configured
> (0.9.0.1):
>
> - https://github.com/apache/storm/blob/v1.0.3/pom.xml#L256-L257
>
> If the problem isn't related to that, then I'd say you are somehow not
> including kafka-clients jar into your uber jar. Can you "jar tf" your uber
> jar and ensure it has the KafkaConsumer class in it? Then you could use
> "jar xf" to get the .class file, and then use "javap" to decompile and look
> for the KafkaConsumer.subscribe() method.
>
> - Erik
>
> On Sat, Feb 25, 2017 at 1:26 PM, pradeep s <sr...@gmail.com>
> wrote:
>
> Hi,
> Tried adding Kafka jar (kafka_2.11) in topology jar dependency as well as
> in storm lib. Still its failing with same error.
> KafkaConsumer class is loaded from kafka-clients-0.10.1.1.jar.
> KafkaConsumer.subscribe call is happening from KafkaSpout which is loaded
> from storm-kafka-client-1.0.3.jar.
>
> *KafkaSpout source code shows below*. Its passing a List of topics to
> subscribe call.
>
> private void subscribeKafkaConsumer() {
>
> kafkaConsumer = new KafkaConsumer<>(kafkaSpoutConfig
> .getKafkaProps(),
>
> kafkaSpoutConfig.getKeyDeserializer(), kafkaSpoutConfig
> .getValueDeserializer());
>
>
> if (kafkaSpoutStreams instanceof KafkaSpoutStreamsNamedTopics) {
>
> * final List<String> topics = ((KafkaSpoutStreamsNamedTopics)
> kafkaSpoutStreams).getTopics();*
>
> kafkaConsumer.subscribe(topics, new
> KafkaSpoutConsumerRebalanceListener());
>
> LOG.info("Kafka consumer subscribed topics {}", topics);
>
> }
>
>
> Verified the worker logs and confirmed that its loading the proper kafka
> version.
>
> 2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser
> Thread-11-merchMariaBolt-executor[102 103] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser
> Thread-11-merchMariaBolt-executor[102 103] [INFO] Kafka commitId :
> f10ef2720b03b247
>
> 2017-02-25 21:20:50.376 o.a.k.c.u.AppInfoParser
> Thread-9-merchMariaBolt-executor[92 93] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
> Thread-9-merchMariaBolt-executor[92 93] [INFO] Kafka commitId :
> f10ef2720b03b247
>
> 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
> Thread-73-merchSpout-executor[432 433] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
> Thread-73-merchSpout-executor[432 433] [INFO] Kafka commitId :
> f10ef2720b03b247
>
> 2017-02-25 21:20:50.381 o.a.k.c.u.AppInfoParser
> Thread-65-merchMariaBolt-executor[32 33] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.382 o.a.k.c.u.AppInfoParser
> Thread-65-merchMariaBolt-executor[32 33] [INFO] Kafka commitId :
> f10ef2720b03b247
>
>
>
> Regards
>
> Pradeep S
>
> On Sat, Feb 25, 2017 at 9:06 AM, Kunal Gupta <ku...@kartrocket.com>
> wrote:
>
> Please add Kafka maven repo in Storm lib directory
>
> On Sat, Feb 25, 2017 at 2:57 PM, pradeep s <sr...@gmail.com>
> wrote:
>
> Hi,
> I am using Storm 1.0.3 and kafka 0.10.1.1 and initialising KafkaSpout
> using storm-kafka-client
> https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md
> Dependencies used are
>
> <dependency>
> <groupId>org.apache.kafka</groupId>
> <artifactId>kafka-clients</artifactId>
> <version>0.10.1.1</version>
> </dependency>
> <dependency>
> <groupId>org.apache.storm</groupId>
> <artifactId>storm-kafka-client</artifactId>
> <version>1.0.3</version>
> </dependency>
>
> While starting the topology spout is failing by below error
> java.lang.*NoSuchMethodError*:
> org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/kafka/clients/consumer/ConsumerRebalanceListener;)V
> at
> org.apache.storm.kafka.spout.KafkaSpout.subscribeKafkaConsumer(KafkaSpout.java:360)
> ~[stormjar.jar:?]
> at
> org.apache.storm.kafka.spout.KafkaSpout.activate(KafkaSpout.java:351)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__4905$fn__4920$fn__4951.invoke(executor.clj:639)
> ~[storm-core-1.0.3.jar:1.0.3]
> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
> [storm-core-1.0.3.jar:1.0.3]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
>
> Regards
> Pradeep S
>
>
>
>
> --
> *Thanks **Kunal*
> *+91-9958189589 <+91%2099581%2089589>*
> *Senior Software Engineer*
>
>
> *Bigfoot Retail Solutions Pvt Ltd.Khasra No. 264, Westend Marg,
> Saidulajab, Saket 110030http://www.kartrocket.com
> <http://www.kartrocket.com/>*
>
>
>
>
Re: Storm kafka spout no such method error
Posted by Erik Weathers <ew...@groupon.com>.
You must be doing *something* wrong.
One thing that stands out to me is that you're using a different version of
the kafka-client (0.10.1.1) than storm-kafka-client has configured
(0.9.0.1):
- https://github.com/apache/storm/blob/v1.0.3/pom.xml#L256-L257
If the problem isn't related to that, then I'd say you are somehow not
including kafka-clients jar into your uber jar. Can you "jar tf" your uber
jar and ensure it has the KafkaConsumer class in it? Then you could use
"jar xf" to get the .class file, and then use "javap" to decompile and look
for the KafkaConsumer.subscribe() method.
- Erik
On Sat, Feb 25, 2017 at 1:26 PM, pradeep s <sr...@gmail.com>
wrote:
> Hi,
> Tried adding Kafka jar (kafka_2.11) in topology jar dependency as well as
> in storm lib. Still its failing with same error.
> KafkaConsumer class is loaded from kafka-clients-0.10.1.1.jar.
> KafkaConsumer.subscribe call is happening from KafkaSpout which is loaded
> from storm-kafka-client-1.0.3.jar.
>
> *KafkaSpout source code shows below*. Its passing a List of topics to
> subscribe call.
>
> private void subscribeKafkaConsumer() {
>
> kafkaConsumer = new KafkaConsumer<>(kafkaSpoutConfig
> .getKafkaProps(),
>
> kafkaSpoutConfig.getKeyDeserializer(), kafkaSpoutConfig.
> getValueDeserializer());
>
>
> if (kafkaSpoutStreams instanceof KafkaSpoutStreamsNamedTopics) {
>
> * final List<String> topics = ((KafkaSpoutStreamsNamedTopics)
> kafkaSpoutStreams).getTopics();*
>
> kafkaConsumer.subscribe(topics, new
> KafkaSpoutConsumerRebalanceListener());
>
> LOG.info("Kafka consumer subscribed topics {}", topics);
>
> }
>
>
> Verified the worker logs and confirmed that its loading the proper kafka
> version.
>
> 2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser Thread-11-merchMariaBolt-executor[102
> 103] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser Thread-11-merchMariaBolt-executor[102
> 103] [INFO] Kafka commitId : f10ef2720b03b247
>
> 2017-02-25 21:20:50.376 o.a.k.c.u.AppInfoParser Thread-9-merchMariaBolt-executor[92
> 93] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser Thread-9-merchMariaBolt-executor[92
> 93] [INFO] Kafka commitId : f10ef2720b03b247
>
> 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
> Thread-73-merchSpout-executor[432 433] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
> Thread-73-merchSpout-executor[432 433] [INFO] Kafka commitId :
> f10ef2720b03b247
>
> 2017-02-25 21:20:50.381 o.a.k.c.u.AppInfoParser Thread-65-merchMariaBolt-executor[32
> 33] [INFO] Kafka version : 0.10.1.1
>
> 2017-02-25 21:20:50.382 o.a.k.c.u.AppInfoParser Thread-65-merchMariaBolt-executor[32
> 33] [INFO] Kafka commitId : f10ef2720b03b247
>
>
>
> Regards
>
> Pradeep S
>
> On Sat, Feb 25, 2017 at 9:06 AM, Kunal Gupta <ku...@kartrocket.com>
> wrote:
>
>> Please add Kafka maven repo in Storm lib directory
>>
>> On Sat, Feb 25, 2017 at 2:57 PM, pradeep s <sr...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I am using Storm 1.0.3 and kafka 0.10.1.1 and initialising KafkaSpout
>>> using storm-kafka-client
>>> https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md
>>> Dependencies used are
>>>
>>> <dependency>
>>> <groupId>org.apache.kafka</groupId>
>>> <artifactId>kafka-clients</artifactId>
>>> <version>0.10.1.1</version>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.storm</groupId>
>>> <artifactId>storm-kafka-client</artifactId>
>>> <version>1.0.3</version>
>>> </dependency>
>>>
>>> While starting the topology spout is failing by below error
>>> java.lang.*NoSuchMethodError*: org.apache.kafka.clients.consu
>>> mer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/kafk
>>> a/clients/consumer/ConsumerRebalanceListener;)V
>>> at org.apache.storm.kafka.spout.KafkaSpout.subscribeKafkaConsumer(KafkaSpout.java:360)
>>> ~[stormjar.jar:?]
>>> at org.apache.storm.kafka.spout.KafkaSpout.activate(KafkaSpout.java:351)
>>> ~[stormjar.jar:?]
>>> at org.apache.storm.daemon.executor$fn__4905$fn__4920$fn__4951.invoke(executor.clj:639)
>>> ~[storm-core-1.0.3.jar:1.0.3]
>>> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
>>> [storm-core-1.0.3.jar:1.0.3]
>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
>>>
>>> Regards
>>> Pradeep S
>>>
>>
>>
>>
>> --
>> *Thanks **Kunal*
>> *+91-9958189589 <+91%2099581%2089589>*
>> *Senior Software Engineer*
>>
>>
>> *Bigfoot Retail Solutions Pvt Ltd.Khasra No. 264, Westend Marg,
>> Saidulajab, Saket 110030http://www.kartrocket.com
>> <http://www.kartrocket.com/>*
>>
>
>
Re: Storm kafka spout no such method error
Posted by pradeep s <sr...@gmail.com>.
Hi,
Tried adding Kafka jar (kafka_2.11) in topology jar dependency as well as
in storm lib. Still its failing with same error.
KafkaConsumer class is loaded from kafka-clients-0.10.1.1.jar.
KafkaConsumer.subscribe call is happening from KafkaSpout which is loaded
from storm-kafka-client-1.0.3.jar.
*KafkaSpout source code shows below*. Its passing a List of topics to
subscribe call.
private void subscribeKafkaConsumer() {
kafkaConsumer = new KafkaConsumer<>(kafkaSpoutConfig
.getKafkaProps(),
kafkaSpoutConfig.getKeyDeserializer(), kafkaSpoutConfig
.getValueDeserializer());
if (kafkaSpoutStreams instanceof KafkaSpoutStreamsNamedTopics) {
* final List<String> topics = ((KafkaSpoutStreamsNamedTopics)
kafkaSpoutStreams).getTopics();*
kafkaConsumer.subscribe(topics, new
KafkaSpoutConsumerRebalanceListener());
LOG.info("Kafka consumer subscribed topics {}", topics);
}
Verified the worker logs and confirmed that its loading the proper kafka
version.
2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser
Thread-11-merchMariaBolt-executor[102 103] [INFO] Kafka version : 0.10.1.1
2017-02-25 21:20:50.373 o.a.k.c.u.AppInfoParser
Thread-11-merchMariaBolt-executor[102 103] [INFO] Kafka commitId :
f10ef2720b03b247
2017-02-25 21:20:50.376 o.a.k.c.u.AppInfoParser
Thread-9-merchMariaBolt-executor[92 93] [INFO] Kafka version : 0.10.1.1
2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
Thread-9-merchMariaBolt-executor[92 93] [INFO] Kafka commitId :
f10ef2720b03b247
2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
Thread-73-merchSpout-executor[432 433] [INFO] Kafka version : 0.10.1.1
2017-02-25 21:20:50.377 o.a.k.c.u.AppInfoParser
Thread-73-merchSpout-executor[432 433] [INFO] Kafka commitId :
f10ef2720b03b247
2017-02-25 21:20:50.381 o.a.k.c.u.AppInfoParser
Thread-65-merchMariaBolt-executor[32 33] [INFO] Kafka version : 0.10.1.1
2017-02-25 21:20:50.382 o.a.k.c.u.AppInfoParser
Thread-65-merchMariaBolt-executor[32 33] [INFO] Kafka commitId :
f10ef2720b03b247
Regards
Pradeep S
On Sat, Feb 25, 2017 at 9:06 AM, Kunal Gupta <ku...@kartrocket.com>
wrote:
> Please add Kafka maven repo in Storm lib directory
>
> On Sat, Feb 25, 2017 at 2:57 PM, pradeep s <sr...@gmail.com>
> wrote:
>
>> Hi,
>> I am using Storm 1.0.3 and kafka 0.10.1.1 and initialising KafkaSpout
>> using storm-kafka-client
>> https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md
>> Dependencies used are
>>
>> <dependency>
>> <groupId>org.apache.kafka</groupId>
>> <artifactId>kafka-clients</artifactId>
>> <version>0.10.1.1</version>
>> </dependency>
>> <dependency>
>> <groupId>org.apache.storm</groupId>
>> <artifactId>storm-kafka-client</artifactId>
>> <version>1.0.3</version>
>> </dependency>
>>
>> While starting the topology spout is failing by below error
>> java.lang.*NoSuchMethodError*: org.apache.kafka.clients.consu
>> mer.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/
>> kafka/clients/consumer/ConsumerRebalanceListener;)V
>> at org.apache.storm.kafka.spout.KafkaSpout.subscribeKafkaConsumer(KafkaSpout.java:360)
>> ~[stormjar.jar:?]
>> at org.apache.storm.kafka.spout.KafkaSpout.activate(KafkaSpout.java:351)
>> ~[stormjar.jar:?]
>> at org.apache.storm.daemon.executor$fn__4905$fn__4920$fn__4951.invoke(executor.clj:639)
>> ~[storm-core-1.0.3.jar:1.0.3]
>> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
>> [storm-core-1.0.3.jar:1.0.3]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
>>
>> Regards
>> Pradeep S
>>
>
>
>
> --
> *Thanks **Kunal*
> *+91-9958189589 <+91%2099581%2089589>*
> *Senior Software Engineer*
>
>
> *Bigfoot Retail Solutions Pvt Ltd.Khasra No. 264, Westend Marg,
> Saidulajab, Saket 110030http://www.kartrocket.com
> <http://www.kartrocket.com/>*
>
Re: Storm kafka spout no such method error
Posted by Kunal Gupta <ku...@kartrocket.com>.
Please add Kafka maven repo in Storm lib directory
On Sat, Feb 25, 2017 at 2:57 PM, pradeep s <sr...@gmail.com>
wrote:
> Hi,
> I am using Storm 1.0.3 and kafka 0.10.1.1 and initialising KafkaSpout
> using storm-kafka-client
> https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md
> Dependencies used are
>
> <dependency>
> <groupId>org.apache.kafka</groupId>
> <artifactId>kafka-clients</artifactId>
> <version>0.10.1.1</version>
> </dependency>
> <dependency>
> <groupId>org.apache.storm</groupId>
> <artifactId>storm-kafka-client</artifactId>
> <version>1.0.3</version>
> </dependency>
>
> While starting the topology spout is failing by below error
> java.lang.*NoSuchMethodError*: org.apache.kafka.clients.
> consumer.KafkaConsumer.subscribe(Ljava/util/List;
> Lorg/apache/kafka/clients/consumer/ConsumerRebalanceListener;)V
> at org.apache.storm.kafka.spout.KafkaSpout.subscribeKafkaConsumer(KafkaSpout.java:360)
> ~[stormjar.jar:?]
> at org.apache.storm.kafka.spout.KafkaSpout.activate(KafkaSpout.java:351)
> ~[stormjar.jar:?]
> at org.apache.storm.daemon.executor$fn__4905$fn__4920$fn__4951.invoke(executor.clj:639)
> ~[storm-core-1.0.3.jar:1.0.3]
> at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
> [storm-core-1.0.3.jar:1.0.3]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
>
> Regards
> Pradeep S
>
--
*Thanks **Kunal*
*+91-9958189589*
*Senior Software Engineer*
*Bigfoot Retail Solutions Pvt Ltd.Khasra No. 264, Westend Marg, Saidulajab,
Saket 110030http://www.kartrocket.com <http://www.kartrocket.com/>*