Storm-Kafka, TransactionalSpout, last version dependencies

[Romain Leroux <le...@gmail.com>]

Hi,

I am currently testing storm 0.9.1-incubating and the only working
KafkaSpout for Kafka 0.8.x that I found was :
<groupId>net.wurstmeister.storm</groupId>
<artifactId>storm-kafka-0.8-plus</artifactId>

Sadly, I have never been able to get the TransactionalSpout working as soon
as I also used some TransactionalState (Cassandra or Memcached) in my
topology.
If someone has some experience with it, I would be pleased to hear that.

Anyway after some tests I finally concluded that something was buggy on the
Storm-Kafka side and tried to switch to the new :
https://github.com/apache/incubator-storm/tree/master/external/storm-kafka

But when I try it locally on a very basic topology I get the following
Exception :

java.lang.NoSuchMethodError:
org.apache.zookeeper.ZooKeeper.<init>(Ljava/lang/String;ILorg/apache/zookeeper/Watcher;Z)V
        at
org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at
org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:169)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at
org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at
org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at
org.apache.curator.ConnectionState.reset(ConnectionState.java:219)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at
org.apache.curator.ConnectionState.start(ConnectionState.java:103)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at
org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:188)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at
org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:234)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at
storm.kafka.DynamicBrokersReader.<init>(DynamicBrokersReader.java:53)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at
storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:41)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:57)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at storm.kafka.trident.Coordinator.<init>(Coordinator.java:33)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at
storm.kafka.trident.TransactionalTridentKafkaSpout.getCoordinator(TransactionalTridentKafkaSpout.java:41)
~[storm-ad-0.0.1-jar-with-dependencies.jar:na]
        at
storm.trident.spout.PartitionedTridentSpoutExecutor$Coordinator.<init>(PartitionedTridentSpoutExecutor.java:47)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
        at
storm.trident.spout.PartitionedTridentSpoutExecutor.getCoordinator(PartitionedTridentSpoutExecutor.java:154)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
        at
storm.trident.topology.MasterBatchCoordinator.open(MasterBatchCoordinator.java:109)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
        at
backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116.invoke(executor.clj:519)
~[na:na]
        at backtype.storm.util$async_loop$fn__390.invoke(util.clj:431)
~[na:na]
        at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.4.0.jar:na]
        at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]

My Zookeeper is 3.4.5 which seems to be the one required for this Spout
(curator 2.4.0)...
Does someone know what settings/dependencies(pom.xml) should be used to
make this Storm-Kafka TransactionalSpout working ?

Re: Storm-Kafka, TransactionalSpout, last version dependencies

[Irek Khasyanov <qu...@gmail.com>]

I’m using storm-plus 0.4 with zookeeper 3.3.6. If you using storm 0.9.1-incubating, you should downgrade storm-plus. Or use storem 0.9.2-incubating which compatible with new zookeeper and curator


— With best regards, Irek Khasyanov