You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Benjamin Cuthbert <cu...@gmail.com> on 2015/05/24 11:26:18 UTC

debugging why storm is not writing to zookeeper

All,

I am having trouble with the spouts not writing data to zookeeper. When I startup storm I see the worker log with

2015-05-24T09:23:36.098+0000 s.k.PartitionManager [WARN] Error reading and/or parsing at ZkNode: /brokers/rawWarehousePriceSpout/partition_2
java.lang.NullPointerException: null
	at storm.kafka.PartitionManager.<init>(PartitionManager.java:76) ~[celertech-analytics-dependencies-DEVELOP-HEAD-SNAPSHOT.jar:na]
	at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) [celertech-analytics-dependencies-DEVELOP-HEAD-SNAPSHOT.jar:na]
	at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) [celertech-analytics-dependencies-DEVELOP-HEAD-SNAPSHOT.jar:na]
	at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:135) [celertech-analytics-dependencies-DEVELOP-HEAD-SNAPSHOT.jar:na]
	at backtype.storm.daemon.executor$fn__4654$fn__4669$fn__4698.invoke(executor.clj:565) [storm-core-0.9.4.jar:0.9.4]
	at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463) [storm-core-0.9.4.jar:0.9.4]
	at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]

I had to create the path manually in zookeeper 

[zk: 127.0.0.1:2181(CONNECTED) 2] ls /brokers/rawWarehousePriceSpout
[partition_1, partition_2, partition_0]
[zk: 127.0.0.1:2181(CONNECTED) 3] ls /brokers/rawWarehousePriceSpout/partition_2
[]
[zk: 127.0.0.1:2181(CONNECTED) 4] get /brokers/rawWarehousePriceSpout/partition_2
{}
cZxid = 0x3981b
ctime = Thu May 21 20:38:49 UTC 2015
mZxid = 0x3981b
mtime = Thu May 21 20:38:49 UTC 2015
pZxid = 0x3981b
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 2
numChildren = 0
[zk: 127.0.0.1:2181(CONNECTED) 5]

But it will never write an offset. How do I debug this?

Re: debugging why storm is not writing to zookeeper

Posted by Benjamin Cuthbert <cu...@gmail.com>.
Interesting my kafka did not create that it just did /brokers

[zk: 127.0.0.1:2181(CONNECTED) 5] ls /
[hbase, zookeeper, admin, consumers, config, hive_zookeeper_namespace_hive, controller, storm, brokers, controller_epoch]
[zk: 127.0.0.1:2181(CONNECTED) 6] ls /brokers
[rawWarehousePriceSpout, topics, ids]
[zk: 127.0.0.1:2181(CONNECTED) 7] ls /kafka
Node does not exist: /kafka
[zk: 127.0.0.1:2181(CONNECTED) 8]

So what should one have in the spout config?

ZkHosts zkhost = new ZkHosts(brokerHosts,"/brokers");  // /brokers -> kafka broker
		SpoutConfig spoutconfig = new SpoutConfig(zkhost, topic, "/brokers", spoutId); 
		


> On 24 May 2015, at 10:47, Binh Nguyen Van <bi...@gmail.com> wrote:
> 
> /kafka

Re: debugging why storm is not writing to zookeeper

Posted by Binh Nguyen Van <bi...@gmail.com>.
Benjamin,

The broker path is the znode that Kafka created in ZooKeeper, it is under
Kafka's root node.
If your Kafka's root not is at /kafka then your broker path is
/kafka/brokers. You should not create those nodes by yourself.

-Binh

On Sun, May 24, 2015 at 2:26 AM, Benjamin Cuthbert <cu...@gmail.com>
wrote:

> All,
>
> I am having trouble with the spouts not writing data to zookeeper. When I
> startup storm I see the worker log with
>
> 2015-05-24T09:23:36.098+0000 s.k.PartitionManager [WARN] Error reading
> and/or parsing at ZkNode: /brokers/rawWarehousePriceSpout/partition_2
> java.lang.NullPointerException: null
>         at storm.kafka.PartitionManager.<init>(PartitionManager.java:76)
> ~[celertech-analytics-dependencies-DEVELOP-HEAD-SNAPSHOT.jar:na]
>         at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98)
> [celertech-analytics-dependencies-DEVELOP-HEAD-SNAPSHOT.jar:na]
>         at
> storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69)
> [celertech-analytics-dependencies-DEVELOP-HEAD-SNAPSHOT.jar:na]
>         at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:135)
> [celertech-analytics-dependencies-DEVELOP-HEAD-SNAPSHOT.jar:na]
>         at
> backtype.storm.daemon.executor$fn__4654$fn__4669$fn__4698.invoke(executor.clj:565)
> [storm-core-0.9.4.jar:0.9.4]
>         at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463)
> [storm-core-0.9.4.jar:0.9.4]
>         at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
>
> I had to create the path manually in zookeeper
>
> [zk: 127.0.0.1:2181(CONNECTED) 2] ls /brokers/rawWarehousePriceSpout
> [partition_1, partition_2, partition_0]
> [zk: 127.0.0.1:2181(CONNECTED) 3] ls
> /brokers/rawWarehousePriceSpout/partition_2
> []
> [zk: 127.0.0.1:2181(CONNECTED) 4] get
> /brokers/rawWarehousePriceSpout/partition_2
> {}
> cZxid = 0x3981b
> ctime = Thu May 21 20:38:49 UTC 2015
> mZxid = 0x3981b
> mtime = Thu May 21 20:38:49 UTC 2015
> pZxid = 0x3981b
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 2
> numChildren = 0
> [zk: 127.0.0.1:2181(CONNECTED) 5]
>
> But it will never write an offset. How do I debug this?