You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Reem Bensimhon <re...@forter.com> on 2014/11/10 11:23:12 UTC
LocalCluster, kill topology and bolt prepares
Hey everybody
We have some bolts with long prepare duration. Some of our tests don’t use all the bolts in the topology. We noticed that in tests which don’t use all the bolts, the unused bolts' prepares are called but the topology doesn’t wait for them to finish before completing the test (meaning an ack, which is fine of course). The issue arises when we try to kill the LocalCluster topology (since the test is done). When we do that Storm raises an unclear exception if a prepare is still in motion:
java.lang.RuntimeException: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.Iterable
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.executor$fn__3352$fn__3364$fn__3411.invoke(executor.clj:746) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__452.invoke(util.clj:431) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
Caused by: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.Iterable
at backtype.storm.util$get_iterator.invoke(util.clj:833) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.executor$mk_task_receiver$fn__3275.invoke(executor.clj:395) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.disruptor$clojure_handler$reify__1437.onEvent(disruptor.clj:58) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 6 common frames omitted
This exception causes our tests to fail. We were looking for something in the lines of waiting for all bolt prepares to be called. Is this possible? Any other ideas that come to mind to get around this issue?
(Storm 0.8.2)
-Re'em
Re: LocalCluster, kill topology and bolt prepares
Posted by Reem Bensimhon <re...@forter.com>.
Ping… (anyone dealt with those InterruptedExceptions before?)
On Nov 10, 2014, at 12:29 PM, Reem Bensimhon <re...@forter.com>> wrote:
Correction: Storm 0.9.2
On Nov 10, 2014, at 12:23 PM, Reem Bensimhon <re...@forter.com>> wrote:
Hey everybody
We have some bolts with long prepare duration. Some of our tests don’t use all the bolts in the topology. We noticed that in tests which don’t use all the bolts, the unused bolts' prepares are called but the topology doesn’t wait for them to finish before completing the test (meaning an ack, which is fine of course). The issue arises when we try to kill the LocalCluster topology (since the test is done). When we do that Storm raises an unclear exception if a prepare is still in motion:
java.lang.RuntimeException: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.Iterable
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.executor$fn__3352$fn__3364$fn__3411.invoke(executor.clj:746) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__452.invoke(util.clj:431) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
Caused by: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.Iterable
at backtype.storm.util$get_iterator.invoke(util.clj:833) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.executor$mk_task_receiver$fn__3275.invoke(executor.clj:395) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.disruptor$clojure_handler$reify__1437.onEvent(disruptor.clj:58) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 6 common frames omitted
This exception causes our tests to fail. We were looking for something in the lines of waiting for all bolt prepares to be called. Is this possible? Any other ideas that come to mind to get around this issue?
(Storm 0.8.2)
-Re'em
Re: LocalCluster, kill topology and bolt prepares
Posted by Reem Bensimhon <re...@forter.com>.
Correction: Storm 0.9.2
On Nov 10, 2014, at 12:23 PM, Reem Bensimhon <re...@forter.com>> wrote:
Hey everybody
We have some bolts with long prepare duration. Some of our tests don’t use all the bolts in the topology. We noticed that in tests which don’t use all the bolts, the unused bolts' prepares are called but the topology doesn’t wait for them to finish before completing the test (meaning an ack, which is fine of course). The issue arises when we try to kill the LocalCluster topology (since the test is done). When we do that Storm raises an unclear exception if a prepare is still in motion:
java.lang.RuntimeException: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.Iterable
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.executor$fn__3352$fn__3364$fn__3411.invoke(executor.clj:746) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__452.invoke(util.clj:431) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72]
Caused by: java.lang.ClassCastException: java.lang.Object cannot be cast to java.lang.Iterable
at backtype.storm.util$get_iterator.invoke(util.clj:833) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.executor$mk_task_receiver$fn__3275.invoke(executor.clj:395) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.disruptor$clojure_handler$reify__1437.onEvent(disruptor.clj:58) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 6 common frames omitted
This exception causes our tests to fail. We were looking for something in the lines of waiting for all bolt prepares to be called. Is this possible? Any other ideas that come to mind to get around this issue?
(Storm 0.8.2)
-Re'em