You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Cuneo, Nicholas" <nc...@tycoint.com> on 2016/11/18 19:10:14 UTC
NullPointerException on startup
Hello,
We are working with storm 1.0.2 and using Kafka client to subscribe to kafka topics to retrieve data. Randomly when our topologies are starting up we receive a null pointer exception which is killing the topology. We can't seem to identify why this happens, or what measures we can take to prevent it. I'm considering raising a ticket with storm to check for null to prevent the topology from crashing.
2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.NullPointerException
at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2]
... 7 more
2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.NullPointerException
at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2]
... 7 more
2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765) [storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_executor_data$fn__7875$fn__7876.invoke(executor.clj:274) [storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
The method and line number in question below:
@Override
public void ack(Object messageId) {
final KafkaSpoutMessageId msgId = (KafkaSpoutMessageId) messageId;
if (!consumerAutoCommitMode) { // Only need to keep track of acked tuples if commits are not done automatically
acked.get(msgId.getTopicPartition()).add(msgId);
}
emitted.remove(msgId);
}
Restarting the topology sometimes fixes the issue but sometimes does not. I don't know if there's some kind of race condition occurring. We have about 15 topologies all connecting to kafka the same way, and this doesn't happen consistently between them.
Does anyone have any insight as to why this occurs or how to prevent it on our end?
Thanks,
Nick Cuneo / Software Engineer, Cloud / Enterprise Software
Tel: +1 949 517 4802 / Mobile: +1 949 243 4952
3 Ada / Irvine, CA 92618 / USA
ncuneo@tycoint.com / www.tyco.com
[cid:image003.png@01D2418C.07311610]
This email (including any attachments) may contain information that is private or business confidential. If you received this email in error, please delete it from your system without copying it and notify sender by reply email so that our records can be corrected
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
Re: NullPointerException on startup
Posted by "Cuneo, Nicholas" <nc...@tycoint.com>.
Nope
---
Sent from Boxer | http://getboxer.com<http://bxr.io/PBID1>
On November 18, 2016 at 11:05:15 PM PST, Harsh Choudhary <sh...@gmail.com> wrote:
If this is the case then it should not happen. As you suspect that 'there’s a race condition where it tries to retrieve the topic partition from the map but it hasn’t been loaded yet' then if Kafka has been up for a long time, then there should not be such problem of loading. Have you modified the Kafka spout?
Cheers!
Harsh Choudhary
On Sat, Nov 19, 2016 at 1:24 AM, Cuneo, Nicholas <nc...@tycoint.com>> wrote:
The spout is initialized during topology submission, so how would you delay that? Kafka is already running for a long period of time.
Thanks,
Nick
---
Sent from Boxer | http://getboxer.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__bxr.io_PBID1&d=DgMFaQ&c=0YGvTs3tT-VMy8_v51yLDw&r=HO8QOrTBfpl5vpyFqqjx7HR-s42tyqk0p44sQ9NHCck&m=zkZ8Y6c5P0EqwI7ZTwOZ3bCYt_ixbyzJQPsaxBWeXmE&s=Jmmf9DivTBkzmqacVYD6102ZQWR9phhPCtnRvDWHy9s&e=>
On November 18, 2016 at 11:51:29 AM PST, Harsh Choudhary <sh...@gmail.com>> wrote:
I think, you may want to start kafka and then give some time for its initialisation and then submit topology.
On Sat, Nov 19, 2016 at 12:55 AM, Cuneo, Nicholas <nc...@tycoint.com>> wrote:
The cause of the null pointer exception is IN kafka spout during an ACK
My guess is there’s a race condition where it tries to retrieve the topic partition from the map but it hasn’t been loaded yet. Which results in a NULL entry during the GET. When the topology starts up it will immediately receive messages but could be slow to startup due to rebalancing of the kafka offsets or other tasks it does when it subscribes.
Like I said, this happens occasionally during startup but not reliably and has nothing to do with my code other than I’m probably acking the received message faster than the kafka spout can finish initialization.
Thanks,
Nick
From: Harsh Choudhary [mailto:shry.harsh@gmail.com<ma...@gmail.com>]
Sent: Friday, November 18, 2016 11:18 AM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: NullPointerException on startup
Hi
This happens when there is some code in bolt or spout which throws Null Pointer Exception. I suggest you to use Debugger in your IDE, to find out where is this happening. You can try making a Local Cluster and runs it in IDE to figure it out easily.
It never happens because of Storm, so do not raise ticket for now.
On Sat, Nov 19, 2016 at 12:40 AM, Cuneo, Nicholas <nc...@tycoint.com>> wrote:
Hello,
We are working with storm 1.0.2 and using Kafka client to subscribe to kafka topics to retrieve data. Randomly when our topologies are starting up we receive a null pointer exception which is killing the topology. We can’t seem to identify why this happens, or what measures we can take to prevent it. I’m considering raising a ticket with storm to check for null to prevent the topology from crashing.
2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.NullPointerException
at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2]
... 7 more
2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.NullPointerException
at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2]
... 7 more
2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765) [storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_executor_data$fn__7875$fn__7876.invoke(executor.clj:274) [storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
The method and line number in question below:
@Override
public void ack(Object messageId) {
final KafkaSpoutMessageId msgId = (KafkaSpoutMessageId) messageId;
if (!consumerAutoCommitMode) { // Only need to keep track of acked tuples if commits are not done automatically
acked.get(msgId.getTopicPartition()).add(msgId);
}
emitted.remove(msgId);
}
Restarting the topology sometimes fixes the issue but sometimes does not. I don’t know if there’s some kind of race condition occurring. We have about 15 topologies all connecting to kafka the same way, and this doesn’t happen consistently between them.
Does anyone have any insight as to why this occurs or how to prevent it on our end?
Thanks,
Nick Cuneo / Software Engineer, Cloud / Enterprise Software
Tel: +1 949 517 4802 / Mobile: +1 949 243 4952
3 Ada / Irvine, CA 92618 / USA
ncuneo@tycoint.com<ma...@tycoint.com> / www.tyco.com<http://www.tyco.com>
[cid:image003.png@01D2418C.07311610]
This email (including any attachments) may contain information that is private or business confidential. If you received this email in error, please delete it from your system without copying it and notify sender by reply email so that our records can be corrected
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
Re: NullPointerException on startup
Posted by Harsh Choudhary <sh...@gmail.com>.
If this is the case then it should not happen. As you suspect that 'there’s
a race condition where it tries to retrieve the topic partition from the
map but it hasn’t been loaded yet' then if Kafka has been up for a long
time, then there should not be such problem of loading. Have you modified
the Kafka spout?
*Cheers!*
Harsh Choudhary
On Sat, Nov 19, 2016 at 1:24 AM, Cuneo, Nicholas <nc...@tycoint.com> wrote:
> The spout is initialized during topology submission, so how would you
> delay that? Kafka is already running for a long period of time.
>
> Thanks,
> Nick
>
>
> ---
> Sent from Boxer | http://getboxer.com <http://bxr.io/PBID1>
>
>
> On November 18, 2016 at 11:51:29 AM PST, Harsh Choudhary <
> shry.harsh@gmail.com> wrote:
>
> I think, you may want to start kafka and then give some time for its
> initialisation and then submit topology.
>
> On Sat, Nov 19, 2016 at 12:55 AM, Cuneo, Nicholas <nc...@tycoint.com>
> wrote:
>
> The cause of the null pointer exception is IN kafka spout during an ACK
>
>
>
> My guess is there’s a race condition where it tries to retrieve the topic
> partition from the map but it hasn’t been loaded yet. Which results in a
> NULL entry during the GET. When the topology starts up it will immediately
> receive messages but could be slow to startup due to rebalancing of the
> kafka offsets or other tasks it does when it subscribes.
>
>
>
> Like I said, this happens occasionally during startup but not reliably and
> has nothing to do with my code other than I’m probably acking the received
> message faster than the kafka spout can finish initialization.
>
>
>
> Thanks,
>
> Nick
>
>
>
> *From:* Harsh Choudhary [mailto:shry.harsh@gmail.com]
> *Sent:* Friday, November 18, 2016 11:18 AM
> *To:* user@storm.apache.org
> *Subject:* Re: NullPointerException on startup
>
>
>
> Hi
>
> This happens when there is some code in bolt or spout which throws Null
> Pointer Exception. I suggest you to use Debugger in your IDE, to find out
> where is this happening. You can try making a Local Cluster and runs it in
> IDE to figure it out easily.
>
> It never happens because of Storm, so do not raise ticket for now.
>
>
>
>
>
>
> On Sat, Nov 19, 2016 at 12:40 AM, Cuneo, Nicholas <nc...@tycoint.com>
> wrote:
>
> Hello,
>
>
>
> We are working with storm 1.0.2 and using Kafka client to subscribe to
> kafka topics to retrieve data. Randomly when our topologies are starting
> up we receive a null pointer exception which is killing the topology. We
> can’t seem to identify why this happens, or what measures we can take to
> prevent it. I’m considering raising a ticket with storm to check for null
> to prevent the topology from crashing.
>
>
>
> 2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died!
>
> java.lang.RuntimeException: java.lang.NullPointerException
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
> [storm-core-1.0.2.jar:1.0.2]
>
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
>
> Caused by: java.lang.NullPointerException
>
> at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316)
> ~[stormjar.jar:?]
>
> at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> ... 7 more
>
> 2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR]
>
> java.lang.RuntimeException: java.lang.NullPointerException
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
> [storm-core-1.0.2.jar:1.0.2]
>
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
>
> Caused by: java.lang.NullPointerException
>
> at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316)
> ~[stormjar.jar:?]
>
> at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> ... 7 more
>
> 2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died")
>
> java.lang.RuntimeException: ("Worker died")
>
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.0.2.jar:1.0.2]
>
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
>
> at org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765)
> [storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$mk_executor_data$fn__7875$
> fn__7876.invoke(executor.clj:274) [storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494)
> [storm-core-1.0.2.jar:1.0.2]
>
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
>
>
>
> The method and line number in question below:
>
>
>
> @Override
>
> public void ack(Object messageId) {
>
> final KafkaSpoutMessageId msgId = (KafkaSpoutMessageId) messageId;
>
> if (!consumerAutoCommitMode) { // Only need to keep track of
> acked tuples if commits are not done automatically
>
> *acked.get(msgId.getTopicPartition()).add(msgId);*
>
> }
>
> emitted.remove(msgId);
>
> }
>
>
>
> Restarting the topology sometimes fixes the issue but sometimes does not.
> I don’t know if there’s some kind of race condition occurring. We have
> about 15 topologies all connecting to kafka the same way, and this doesn’t
> happen consistently between them.
>
>
>
> Does anyone have any insight as to why this occurs or how to prevent it on
> our end?
>
>
>
>
>
> Thanks,
>
>
>
> *Nick Cuneo* */ *Software Engineer, Cloud */ Enterprise Software*
>
> Tel: +1 949 517 4802 / Mobile: +1 949 243 4952
> 3 Ada / Irvine, CA 92618 / USA
>
> *ncuneo@tycoint.com <nc...@tycoint.com>* / *www.tyco.com
> <http://www.tyco.com>*
>
> [image: cid:image003.png@01D2418C.07311610]
>
>
>
> This email (including any attachments) may contain information that is
> private or business confidential. If you received this email in error,
> please delete it from your system without copying it and notify sender by
> reply email so that our records can be corrected
>
>
>
>
> ------------------------------
>
>
> This e-mail contains privileged and confidential information intended for
> the use of the addressees named above. If you are not the intended
> recipient of this e-mail, you are hereby notified that you must not
> disseminate, copy or take any action in respect of any information
> contained in it. If you have received this e-mail in error, please notify
> the sender immediately by e-mail and immediately destroy this e-mail and
> its attachments.
>
>
>
> ------------------------------
>
> This e-mail contains privileged and confidential information intended for
> the use of the addressees named above. If you are not the intended
> recipient of this e-mail, you are hereby notified that you must not
> disseminate, copy or take any action in respect of any information
> contained in it. If you have received this e-mail in error, please notify
> the sender immediately by e-mail and immediately destroy this e-mail and
> its attachments.
>
>
>
> ------------------------------
>
> This e-mail contains privileged and confidential information intended for
> the use of the addressees named above. If you are not the intended
> recipient of this e-mail, you are hereby notified that you must not
> disseminate, copy or take any action in respect of any information
> contained in it. If you have received this e-mail in error, please notify
> the sender immediately by e-mail and immediately destroy this e-mail and
> its attachments.
>
Re: NullPointerException on startup
Posted by "Cuneo, Nicholas" <nc...@tycoint.com>.
The spout is initialized during topology submission, so how would you delay that? Kafka is already running for a long period of time.
Thanks,
Nick
---
Sent from Boxer | http://getboxer.com<http://bxr.io/PBID1>
On November 18, 2016 at 11:51:29 AM PST, Harsh Choudhary <sh...@gmail.com> wrote:
I think, you may want to start kafka and then give some time for its initialisation and then submit topology.
On Sat, Nov 19, 2016 at 12:55 AM, Cuneo, Nicholas <nc...@tycoint.com>> wrote:
The cause of the null pointer exception is IN kafka spout during an ACK
My guess is there’s a race condition where it tries to retrieve the topic partition from the map but it hasn’t been loaded yet. Which results in a NULL entry during the GET. When the topology starts up it will immediately receive messages but could be slow to startup due to rebalancing of the kafka offsets or other tasks it does when it subscribes.
Like I said, this happens occasionally during startup but not reliably and has nothing to do with my code other than I’m probably acking the received message faster than the kafka spout can finish initialization.
Thanks,
Nick
From: Harsh Choudhary [mailto:shry.harsh@gmail.com<ma...@gmail.com>]
Sent: Friday, November 18, 2016 11:18 AM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: NullPointerException on startup
Hi
This happens when there is some code in bolt or spout which throws Null Pointer Exception. I suggest you to use Debugger in your IDE, to find out where is this happening. You can try making a Local Cluster and runs it in IDE to figure it out easily.
It never happens because of Storm, so do not raise ticket for now.
On Sat, Nov 19, 2016 at 12:40 AM, Cuneo, Nicholas <nc...@tycoint.com>> wrote:
Hello,
We are working with storm 1.0.2 and using Kafka client to subscribe to kafka topics to retrieve data. Randomly when our topologies are starting up we receive a null pointer exception which is killing the topology. We can’t seem to identify why this happens, or what measures we can take to prevent it. I’m considering raising a ticket with storm to check for null to prevent the topology from crashing.
2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.NullPointerException
at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2]
... 7 more
2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.NullPointerException
at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2]
... 7 more
2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765) [storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_executor_data$fn__7875$fn__7876.invoke(executor.clj:274) [storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
The method and line number in question below:
@Override
public void ack(Object messageId) {
final KafkaSpoutMessageId msgId = (KafkaSpoutMessageId) messageId;
if (!consumerAutoCommitMode) { // Only need to keep track of acked tuples if commits are not done automatically
acked.get(msgId.getTopicPartition()).add(msgId);
}
emitted.remove(msgId);
}
Restarting the topology sometimes fixes the issue but sometimes does not. I don’t know if there’s some kind of race condition occurring. We have about 15 topologies all connecting to kafka the same way, and this doesn’t happen consistently between them.
Does anyone have any insight as to why this occurs or how to prevent it on our end?
Thanks,
Nick Cuneo / Software Engineer, Cloud / Enterprise Software
Tel: +1 949 517 4802 / Mobile: +1 949 243 4952
3 Ada / Irvine, CA 92618 / USA
ncuneo@tycoint.com<ma...@tycoint.com> / www.tyco.com<http://www.tyco.com>
[cid:image003.png@01D2418C.07311610]
This email (including any attachments) may contain information that is private or business confidential. If you received this email in error, please delete it from your system without copying it and notify sender by reply email so that our records can be corrected
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
Re: NullPointerException on startup
Posted by Harsh Choudhary <sh...@gmail.com>.
I think, you may want to start kafka and then give some time for its
initialisation and then submit topology.
On Sat, Nov 19, 2016 at 12:55 AM, Cuneo, Nicholas <nc...@tycoint.com>
wrote:
> The cause of the null pointer exception is IN kafka spout during an ACK
>
>
>
> My guess is there’s a race condition where it tries to retrieve the topic
> partition from the map but it hasn’t been loaded yet. Which results in a
> NULL entry during the GET. When the topology starts up it will immediately
> receive messages but could be slow to startup due to rebalancing of the
> kafka offsets or other tasks it does when it subscribes.
>
>
>
> Like I said, this happens occasionally during startup but not reliably and
> has nothing to do with my code other than I’m probably acking the received
> message faster than the kafka spout can finish initialization.
>
>
>
> Thanks,
>
> Nick
>
>
>
> *From:* Harsh Choudhary [mailto:shry.harsh@gmail.com]
> *Sent:* Friday, November 18, 2016 11:18 AM
> *To:* user@storm.apache.org
> *Subject:* Re: NullPointerException on startup
>
>
>
> Hi
>
> This happens when there is some code in bolt or spout which throws Null
> Pointer Exception. I suggest you to use Debugger in your IDE, to find out
> where is this happening. You can try making a Local Cluster and runs it in
> IDE to figure it out easily.
>
> It never happens because of Storm, so do not raise ticket for now.
>
>
>
>
>
>
> On Sat, Nov 19, 2016 at 12:40 AM, Cuneo, Nicholas <nc...@tycoint.com>
> wrote:
>
> Hello,
>
>
>
> We are working with storm 1.0.2 and using Kafka client to subscribe to
> kafka topics to retrieve data. Randomly when our topologies are starting
> up we receive a null pointer exception which is killing the topology. We
> can’t seem to identify why this happens, or what measures we can take to
> prevent it. I’m considering raising a ticket with storm to check for null
> to prevent the topology from crashing.
>
>
>
> 2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died!
>
> java.lang.RuntimeException: java.lang.NullPointerException
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
> [storm-core-1.0.2.jar:1.0.2]
>
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
>
> Caused by: java.lang.NullPointerException
>
> at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316)
> ~[stormjar.jar:?]
>
> at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$fn__7990$tuple_
> action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> ... 7 more
>
> 2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR]
>
> java.lang.RuntimeException: java.lang.NullPointerException
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
> [storm-core-1.0.2.jar:1.0.2]
>
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
>
> Caused by: java.lang.NullPointerException
>
> at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316)
> ~[stormjar.jar:?]
>
> at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$fn__7990$tuple_
> action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> ... 7 more
>
> 2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died")
>
> java.lang.RuntimeException: ("Worker died")
>
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.0.2.jar:1.0.2]
>
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
>
> at org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765)
> [storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$mk_executor_data$fn__
> 7875$fn__7876.invoke(executor.clj:274) [storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494)
> [storm-core-1.0.2.jar:1.0.2]
>
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
>
>
>
> The method and line number in question below:
>
>
>
> @Override
>
> public void ack(Object messageId) {
>
> final KafkaSpoutMessageId msgId = (KafkaSpoutMessageId) messageId;
>
> if (!consumerAutoCommitMode) { // Only need to keep track of
> acked tuples if commits are not done automatically
>
> *acked.get(msgId.getTopicPartition()).add(msgId);*
>
> }
>
> emitted.remove(msgId);
>
> }
>
>
>
> Restarting the topology sometimes fixes the issue but sometimes does not.
> I don’t know if there’s some kind of race condition occurring. We have
> about 15 topologies all connecting to kafka the same way, and this doesn’t
> happen consistently between them.
>
>
>
> Does anyone have any insight as to why this occurs or how to prevent it on
> our end?
>
>
>
>
>
> Thanks,
>
>
>
> *Nick Cuneo* */ *Software Engineer, Cloud */ Enterprise Software*
>
> Tel: +1 949 517 4802 / Mobile: +1 949 243 4952
> 3 Ada / Irvine, CA 92618 / USA
>
> *ncuneo@tycoint.com <nc...@tycoint.com>* / *www.tyco.com
> <http://www.tyco.com>*
>
> [image: cid:image003.png@01D2418C.07311610]
>
>
>
> This email (including any attachments) may contain information that is
> private or business confidential. If you received this email in error,
> please delete it from your system without copying it and notify sender by
> reply email so that our records can be corrected
>
>
>
>
> ------------------------------
>
>
> This e-mail contains privileged and confidential information intended for
> the use of the addressees named above. If you are not the intended
> recipient of this e-mail, you are hereby notified that you must not
> disseminate, copy or take any action in respect of any information
> contained in it. If you have received this e-mail in error, please notify
> the sender immediately by e-mail and immediately destroy this e-mail and
> its attachments.
>
>
>
> ------------------------------
>
> This e-mail contains privileged and confidential information intended for
> the use of the addressees named above. If you are not the intended
> recipient of this e-mail, you are hereby notified that you must not
> disseminate, copy or take any action in respect of any information
> contained in it. If you have received this e-mail in error, please notify
> the sender immediately by e-mail and immediately destroy this e-mail and
> its attachments.
>
RE: NullPointerException on startup
Posted by "Cuneo, Nicholas" <nc...@tycoint.com>.
The cause of the null pointer exception is IN kafka spout during an ACK
My guess is there’s a race condition where it tries to retrieve the topic partition from the map but it hasn’t been loaded yet. Which results in a NULL entry during the GET. When the topology starts up it will immediately receive messages but could be slow to startup due to rebalancing of the kafka offsets or other tasks it does when it subscribes.
Like I said, this happens occasionally during startup but not reliably and has nothing to do with my code other than I’m probably acking the received message faster than the kafka spout can finish initialization.
Thanks,
Nick
From: Harsh Choudhary [mailto:shry.harsh@gmail.com]
Sent: Friday, November 18, 2016 11:18 AM
To: user@storm.apache.org
Subject: Re: NullPointerException on startup
Hi
This happens when there is some code in bolt or spout which throws Null Pointer Exception. I suggest you to use Debugger in your IDE, to find out where is this happening. You can try making a Local Cluster and runs it in IDE to figure it out easily.
It never happens because of Storm, so do not raise ticket for now.
On Sat, Nov 19, 2016 at 12:40 AM, Cuneo, Nicholas <nc...@tycoint.com>> wrote:
Hello,
We are working with storm 1.0.2 and using Kafka client to subscribe to kafka topics to retrieve data. Randomly when our topologies are starting up we receive a null pointer exception which is killing the topology. We can’t seem to identify why this happens, or what measures we can take to prevent it. I’m considering raising a ticket with storm to check for null to prevent the topology from crashing.
2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.NullPointerException
at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2]
... 7 more
2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.NullPointerException
at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2]
... 7 more
2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765) [storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_executor_data$fn__7875$fn__7876.invoke(executor.clj:274) [storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
The method and line number in question below:
@Override
public void ack(Object messageId) {
final KafkaSpoutMessageId msgId = (KafkaSpoutMessageId) messageId;
if (!consumerAutoCommitMode) { // Only need to keep track of acked tuples if commits are not done automatically
acked.get(msgId.getTopicPartition()).add(msgId);
}
emitted.remove(msgId);
}
Restarting the topology sometimes fixes the issue but sometimes does not. I don’t know if there’s some kind of race condition occurring. We have about 15 topologies all connecting to kafka the same way, and this doesn’t happen consistently between them.
Does anyone have any insight as to why this occurs or how to prevent it on our end?
Thanks,
Nick Cuneo / Software Engineer, Cloud / Enterprise Software
Tel: +1 949 517 4802 / Mobile: +1 949 243 4952
3 Ada / Irvine, CA 92618 / USA
ncuneo@tycoint.com<ma...@tycoint.com> / www.tyco.com<http://www.tyco.com>
[cid:image003.png@01D2418C.07311610]
This email (including any attachments) may contain information that is private or business confidential. If you received this email in error, please delete it from your system without copying it and notify sender by reply email so that our records can be corrected
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
Re: NullPointerException on startup
Posted by Harsh Choudhary <sh...@gmail.com>.
Hi
This happens when there is some code in bolt or spout which throws Null
Pointer Exception. I suggest you to use Debugger in your IDE, to find out
where is this happening. You can try making a Local Cluster and runs it in
IDE to figure it out easily.
It never happens because of Storm, so do not raise ticket for now.
On Sat, Nov 19, 2016 at 12:40 AM, Cuneo, Nicholas <nc...@tycoint.com>
wrote:
> Hello,
>
>
>
> We are working with storm 1.0.2 and using Kafka client to subscribe to
> kafka topics to retrieve data. Randomly when our topologies are starting
> up we receive a null pointer exception which is killing the topology. We
> can’t seem to identify why this happens, or what measures we can take to
> prevent it. I’m considering raising a ticket with storm to check for null
> to prevent the topology from crashing.
>
>
>
> 2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died!
>
> java.lang.RuntimeException: java.lang.NullPointerException
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
> [storm-core-1.0.2.jar:1.0.2]
>
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
>
> Caused by: java.lang.NullPointerException
>
> at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316)
> ~[stormjar.jar:?]
>
> at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$fn__7990$tuple_
> action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> ... 7 more
>
> 2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR]
>
> java.lang.RuntimeException: java.lang.NullPointerException
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484)
> [storm-core-1.0.2.jar:1.0.2]
>
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
>
> Caused by: java.lang.NullPointerException
>
> at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316)
> ~[stormjar.jar:?]
>
> at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$fn__7990$tuple_
> action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> ~[storm-core-1.0.2.jar:1.0.2]
>
> ... 7 more
>
> 2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died")
>
> java.lang.RuntimeException: ("Worker died")
>
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.0.2.jar:1.0.2]
>
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
>
> at org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765)
> [storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.daemon.executor$mk_executor_data$fn__
> 7875$fn__7876.invoke(executor.clj:274) [storm-core-1.0.2.jar:1.0.2]
>
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494)
> [storm-core-1.0.2.jar:1.0.2]
>
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
>
>
>
> The method and line number in question below:
>
>
>
> @Override
>
> public void ack(Object messageId) {
>
> final KafkaSpoutMessageId msgId = (KafkaSpoutMessageId) messageId;
>
> if (!consumerAutoCommitMode) { // Only need to keep track of
> acked tuples if commits are not done automatically
>
> *acked.get(msgId.getTopicPartition()).add(msgId);*
>
> }
>
> emitted.remove(msgId);
>
> }
>
>
>
> Restarting the topology sometimes fixes the issue but sometimes does not.
> I don’t know if there’s some kind of race condition occurring. We have
> about 15 topologies all connecting to kafka the same way, and this doesn’t
> happen consistently between them.
>
>
>
> Does anyone have any insight as to why this occurs or how to prevent it on
> our end?
>
>
>
>
>
> Thanks,
>
>
>
> *Nick Cuneo* */ *Software Engineer, Cloud */ Enterprise Software*
>
> Tel: +1 949 517 4802 / Mobile: +1 949 243 4952
> 3 Ada / Irvine, CA 92618 / USA
>
> *ncuneo@tycoint.com <nc...@tycoint.com>* / *www.tyco.com
> <http://www.tyco.com>*
>
> [image: cid:image003.png@01D2418C.07311610]
>
>
>
> This email (including any attachments) may contain information that is
> private or business confidential. If you received this email in error,
> please delete it from your system without copying it and notify sender by
> reply email so that our records can be corrected
>
>
>
> ------------------------------
>
> This e-mail contains privileged and confidential information intended for
> the use of the addressees named above. If you are not the intended
> recipient of this e-mail, you are hereby notified that you must not
> disseminate, copy or take any action in respect of any information
> contained in it. If you have received this e-mail in error, please notify
> the sender immediately by e-mail and immediately destroy this e-mail and
> its attachments.
>
RE: NullPointerException on startup
Posted by "Cuneo, Nicholas" <nc...@tycoint.com>.
I’m having trouble finding that version, is it located in a separate repository?
Thanks,
Nick
From: P. Taylor Goetz [mailto:ptgoetz@gmail.com]
Sent: Friday, November 18, 2016 12:40 PM
To: dev@storm.apache.org; Cuneo, Nicholas
Subject: Re: NullPointerException on startup
(Moving thread to dev@ to discuss versions not yet released)
Nick,
I suspect you might be hitting a bug in 1.0.2 that might be fixed in one of the next releases.
Can you try setting the version of storm-kafka-client to 1.1.0-SNAPSHOT (leave other storm components at 1.0.2)?
If that fixes the issue, let us know on dev@storm.apache.org<ma...@storm.apache.org> so we can back port the fix.
-Taylor
On Nov 18, 2016, at 2:10 PM, Cuneo, Nicholas <nc...@tycoint.com>> wrote:
Hello,
We are working with storm 1.0.2 and using Kafka client to subscribe to kafka topics to retrieve data. Randomly when our topologies are starting up we receive a null pointer exception which is killing the topology. We can’t seem to identify why this happens, or what measures we can take to prevent it. I’m considering raising a ticket with storm to check for null to prevent the topology from crashing.
2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.NullPointerException
at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2]
... 7 more
2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.lang.NullPointerException
at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) ~[stormjar.jar:?]
at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2]
... 7 more
2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765) [storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.daemon.executor$mk_executor_data$fn__7875$fn__7876.invoke(executor.clj:274) [storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.0.2.jar:1.0.2]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
The method and line number in question below:
@Override
public void ack(Object messageId) {
final KafkaSpoutMessageId msgId = (KafkaSpoutMessageId) messageId;
if (!consumerAutoCommitMode) { // Only need to keep track of acked tuples if commits are not done automatically
acked.get(msgId.getTopicPartition()).add(msgId);
}
emitted.remove(msgId);
}
Restarting the topology sometimes fixes the issue but sometimes does not. I don’t know if there’s some kind of race condition occurring. We have about 15 topologies all connecting to kafka the same way, and this doesn’t happen consistently between them.
Does anyone have any insight as to why this occurs or how to prevent it on our end?
Thanks,
Nick Cuneo / Software Engineer, Cloud / Enterprise Software
Tel: +1 949 517 4802 / Mobile: +1 949 243 4952
3 Ada / Irvine, CA 92618 / USA
ncuneo@tycoint.com<ma...@tycoint.com> / www.tyco.com<http://www.tyco.com/>
<image003.png>
This email (including any attachments) may contain information that is private or business confidential. If you received this email in error, please delete it from your system without copying it and notify sender by reply email so that our records can be corrected
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
________________________________
This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
Re: NullPointerException on startup
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
(Moving thread to dev@ to discuss versions not yet released)
Nick,
I suspect you might be hitting a bug in 1.0.2 that might be fixed in one of the next releases.
Can you try setting the version of storm-kafka-client to 1.1.0-SNAPSHOT (leave other storm components at 1.0.2)?
If that fixes the issue, let us know on dev@storm.apache.org <ma...@storm.apache.org> so we can back port the fix.
-Taylor
> On Nov 18, 2016, at 2:10 PM, Cuneo, Nicholas <nc...@tycoint.com> wrote:
>
> Hello,
>
> We are working with storm 1.0.2 and using Kafka client to subscribe to kafka topics to retrieve data. Randomly when our topologies are starting up we receive a null pointer exception which is killing the topology. We can’t seem to identify why this happens, or what measures we can take to prevent it. I’m considering raising a ticket with storm to check for null to prevent the topology from crashing.
>
> 2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died!
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
> Caused by: java.lang.NullPointerException
> at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) ~[stormjar.jar:?]
> at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2]
> ... 7 more
> 2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR]
> java.lang.RuntimeException: java.lang.NullPointerException
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
> Caused by: java.lang.NullPointerException
> at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) ~[stormjar.jar:?]
> at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2]
> ... 7 more
> 2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) [storm-core-1.0.2.jar:1.0.2]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765) [storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.daemon.executor$mk_executor_data$fn__7875$fn__7876.invoke(executor.clj:274) [storm-core-1.0.2.jar:1.0.2]
> at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) [storm-core-1.0.2.jar:1.0.2]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
>
> The method and line number in question below:
>
> @Override
> public void ack(Object messageId) {
> final KafkaSpoutMessageId msgId = (KafkaSpoutMessageId) messageId;
> if (!consumerAutoCommitMode) { // Only need to keep track of acked tuples if commits are not done automatically
> acked.get(msgId.getTopicPartition()).add(msgId);
> }
> emitted.remove(msgId);
> }
>
> Restarting the topology sometimes fixes the issue but sometimes does not. I don’t know if there’s some kind of race condition occurring. We have about 15 topologies all connecting to kafka the same way, and this doesn’t happen consistently between them.
>
> Does anyone have any insight as to why this occurs or how to prevent it on our end?
>
>
> Thanks,
>
> Nick Cuneo / Software Engineer, Cloud / Enterprise Software
> Tel: +1 949 517 4802 / Mobile: +1 949 243 4952
> 3 Ada / Irvine, CA 92618 / USA
> ncuneo@tycoint.com <ma...@tycoint.com> / www.tyco.com <http://www.tyco.com/>
> <image003.png>
>
>
> This email (including any attachments) may contain information that is private or business confidential. If you received this email in error, please delete it from your system without copying it and notify sender by reply email so that our records can be corrected
>
>
>
> This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.