You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Asif Ihsan <as...@gmail.com> on 2015/05/13 09:37:48 UTC
Multiple Workers in local mode gives error
Hi,
I recently started using Storm. I am having topology. It works well with
numofworkers=1. However when I change it to 6 or more than 1. Its gives
error when emitting tuples. What is the issue and how to resolve it with
multiple workers. Thank you.
--
Regards
Muhammad Asif Ihsan
RE: Multiple Workers in local mode gives error
Posted by Ra...@DellTeam.com.
Dell - Internal Use - Confidential
Assuming your parallelism hints default to 1, you have combined parallelism of 6 and with 6 workers each would get 1 executor each.
So each of the bolt and task will run on a different worker(if you have >6 available slots on your cluster)
Checkout the awesome docs @ https://storm.apache.org/documentation/Understanding-the-parallelism-of-a-Storm-topology.html
From: Asif Ihsan [mailto:asifihsan.ihsan@gmail.com]
Sent: Thursday, May 14, 2015 9:47 AM
To: user@storm.apache.org
Subject: Re: Multiple Workers in local mode gives error
If I create 6 workers for 5 bolts and 1 spout. Then how will these spout and bolts be distributed among workers.
On Thu, May 14, 2015 at 5:34 PM, 임정택 <ka...@gmail.com>> wrote:
Yes, right.
Worker is a JVM instance which can contains one or more executors, each executor is spout, bolt, acker, etc.
2015년 5월 14일 목요일, Asif Ihsan<as...@gmail.com>>님이 작성한 메시지:
You are right about it. What does multiple workers means. Does it mean that when I run topology with single worker than i will see one single process handling all the bolts and when running 6 workers for 5 bolts and 1 spout than i will see 6 separate processes running each bolt and spout? M i right.If not then what does it mean?
On Thu, May 14, 2015 at 2:46 PM, 임정택 <ka...@gmail.com>> wrote:
Hi.
Storm serializes tuples when tuple should be sent to other (remote) worker.
In other words, Storm doesn't serialize tuples when destination is local task. That's why you didn't meet error when testing with 1 worker.
MapEventBean seems to be not serializable, so you need to convert to other data type.
Hope this helps.
Regards.
Jungtaek Lim (HeartSaVioR)
2015-05-14 18:38 GMT+09:00 Asif Ihsan <as...@gmail.com>>:
I am using Esper in one of the bolts of the Storm topology. Bolt emit MapEventBean array (EventBean[]). With the single worker topology run smoothly. With multiple workers it gives following error in emit function call.
May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
SEVERE: Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
at backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
at backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
at backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
at backtype.storm.util$fast_list_map.invoke(util.clj:771)
at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
at backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
... 6 more
Caused by: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeArray(Unknown Source)
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeObject(Unknown Source)
at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
... 21 more
May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
SEVERE:
java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
at backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
at backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
at backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
at backtype.storm.util$fast_list_map.invoke(util.clj:771)
at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
at backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
... 6 more
Caused by: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeArray(Unknown Source)
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeObject(Unknown Source)
at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
... 21 more
On Wed, May 13, 2015 at 6:11 PM, Jeffery Maass <ma...@gmail.com>> wrote:
Asif:
Would love to help you, however, we will need way more details. Why don't you start with telling us what the error is? How are you detecting the error?
You will want to look over both the supervisor and worker logs.
Thank you for your time!
+++++++++++++++++++++
Jeff Maass
linkedin.com/in/jeffmaass<http://linkedin.com/in/jeffmaass>
stackoverflow.com/users/373418/maassql<http://stackoverflow.com/users/373418/maassql>
+++++++++++++++++++++
On Wed, May 13, 2015 at 2:37 AM, Asif Ihsan <as...@gmail.com>> wrote:
Hi,
I recently started using Storm. I am having topology. It works well with numofworkers=1. However when I change it to 6 or more than 1. Its gives error when emitting tuples. What is the issue and how to resolve it with multiple workers. Thank you.
--
Regards
Muhammad Asif Ihsan
--
Regards
Muhammad Asif Ihsan
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
--
Regards
Muhammad Asif Ihsan
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
--
Regards
Muhammad Asif Ihsan
Re: Multiple Workers in local mode gives error
Posted by Asif Ihsan <as...@gmail.com>.
If I create 6 workers for 5 bolts and 1 spout. Then how will these spout
and bolts be distributed among workers.
On Thu, May 14, 2015 at 5:34 PM, 임정택 <ka...@gmail.com> wrote:
> Yes, right.
> Worker is a JVM instance which can contains one or more executors, each
> executor is spout, bolt, acker, etc.
>
> 2015년 5월 14일 목요일, Asif Ihsan<as...@gmail.com>님이 작성한 메시지:
>
> You are right about it. What does multiple workers means. Does it mean
>> that when I run topology with single worker than i will see one single
>> process handling all the bolts and when running 6 workers for 5 bolts and 1
>> spout than i will see 6 separate processes running each bolt and spout? M i
>> right.If not then what does it mean?
>>
>>
>> On Thu, May 14, 2015 at 2:46 PM, 임정택 <ka...@gmail.com> wrote:
>>
>>> Hi.
>>>
>>> Storm serializes tuples when tuple should be sent to other (remote)
>>> worker.
>>> In other words, Storm doesn't serialize tuples when destination is local
>>> task. That's why you didn't meet error when testing with 1 worker.
>>>
>>> MapEventBean seems to be not serializable, so you need to convert to
>>> other data type.
>>>
>>> Hope this helps.
>>>
>>> Regards.
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>>
>>>
>>> 2015-05-14 18:38 GMT+09:00 Asif Ihsan <as...@gmail.com>:
>>>
>>>> I am using Esper in one of the bolts of the Storm topology. Bolt emit
>>>> MapEventBean array (EventBean[]). With the single worker topology run
>>>> smoothly. With multiple workers it gives following error in emit function
>>>> call.
>>>>
>>>> May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
>>>> SEVERE: Async loop died!
>>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>>> java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
>>>> at
>>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
>>>> at
>>>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
>>>> at
>>>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
>>>> at
>>>> backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
>>>> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
>>>> at clojure.lang.AFn.run(AFn.java:24)
>>>> at java.lang.Thread.run(Unknown Source)
>>>> Caused by: java.lang.RuntimeException:
>>>> java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
>>>> at
>>>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
>>>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>>>> at
>>>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
>>>> at
>>>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
>>>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>>>> at
>>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
>>>> at
>>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
>>>> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
>>>> at
>>>> backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
>>>> at
>>>> backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
>>>> at
>>>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
>>>> at backtype.storm.util$fast_list_map.invoke(util.clj:771)
>>>> at
>>>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
>>>> at
>>>> backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
>>>> at
>>>> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
>>>> at
>>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
>>>> ... 6 more
>>>> Caused by: java.io.NotSerializableException:
>>>> com.espertech.esper.event.map.MapEventBean
>>>> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>>>> at java.io.ObjectOutputStream.writeArray(Unknown Source)
>>>> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>>>> at java.io.ObjectOutputStream.writeObject(Unknown Source)
>>>> at
>>>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
>>>> ... 21 more
>>>>
>>>> May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
>>>> SEVERE:
>>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>>> java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
>>>> at
>>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
>>>> at
>>>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
>>>> at
>>>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
>>>> at
>>>> backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
>>>> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
>>>> at clojure.lang.AFn.run(AFn.java:24)
>>>> at java.lang.Thread.run(Unknown Source)
>>>> Caused by: java.lang.RuntimeException:
>>>> java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
>>>> at
>>>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
>>>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>>>> at
>>>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
>>>> at
>>>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
>>>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>>>> at
>>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
>>>> at
>>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
>>>> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
>>>> at
>>>> backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
>>>> at
>>>> backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
>>>> at
>>>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
>>>> at backtype.storm.util$fast_list_map.invoke(util.clj:771)
>>>> at
>>>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
>>>> at
>>>> backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
>>>> at
>>>> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
>>>> at
>>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
>>>> ... 6 more
>>>> Caused by: java.io.NotSerializableException:
>>>> com.espertech.esper.event.map.MapEventBean
>>>> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>>>> at java.io.ObjectOutputStream.writeArray(Unknown Source)
>>>> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>>>> at java.io.ObjectOutputStream.writeObject(Unknown Source)
>>>> at
>>>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
>>>> ... 21 more
>>>>
>>>>
>>>> On Wed, May 13, 2015 at 6:11 PM, Jeffery Maass <ma...@gmail.com>
>>>> wrote:
>>>>
>>>>> Asif:
>>>>>
>>>>> Would love to help you, however, we will need way more details. Why
>>>>> don't you start with telling us what the error is? How are you detecting
>>>>> the error?
>>>>>
>>>>> You will want to look over both the supervisor and worker logs.
>>>>>
>>>>> Thank you for your time!
>>>>>
>>>>> +++++++++++++++++++++
>>>>> Jeff Maass
>>>>> linkedin.com/in/jeffmaass
>>>>> stackoverflow.com/users/373418/maassql
>>>>> +++++++++++++++++++++
>>>>>
>>>>>
>>>>> On Wed, May 13, 2015 at 2:37 AM, Asif Ihsan <asifihsan.ihsan@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I recently started using Storm. I am having topology. It works well
>>>>>> with numofworkers=1. However when I change it to 6 or more than 1. Its
>>>>>> gives error when emitting tuples. What is the issue and how to resolve it
>>>>>> with multiple workers. Thank you.
>>>>>>
>>>>>> --
>>>>>> Regards
>>>>>> Muhammad Asif Ihsan
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards
>>>> Muhammad Asif Ihsan
>>>>
>>>
>>>
>>>
>>> --
>>> Name : 임 정택
>>> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
>>> Twitter : http://twitter.com/heartsavior
>>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>>
>>
>>
>>
>> --
>> Regards
>> Muhammad Asif Ihsan
>>
>
>
> --
> Name : 임 정택
> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>
>
--
Regards
Muhammad Asif Ihsan
Re: Multiple Workers in local mode gives error
Posted by 임정택 <ka...@gmail.com>.
Yes, right.
Worker is a JVM instance which can contains one or more executors, each
executor is spout, bolt, acker, etc.
2015년 5월 14일 목요일, Asif Ihsan<as...@gmail.com>님이 작성한 메시지:
> You are right about it. What does multiple workers means. Does it mean
> that when I run topology with single worker than i will see one single
> process handling all the bolts and when running 6 workers for 5 bolts and 1
> spout than i will see 6 separate processes running each bolt and spout? M i
> right.If not then what does it mean?
>
>
> On Thu, May 14, 2015 at 2:46 PM, 임정택 <kabhwan@gmail.com
> <javascript:_e(%7B%7D,'cvml','kabhwan@gmail.com');>> wrote:
>
>> Hi.
>>
>> Storm serializes tuples when tuple should be sent to other (remote)
>> worker.
>> In other words, Storm doesn't serialize tuples when destination is local
>> task. That's why you didn't meet error when testing with 1 worker.
>>
>> MapEventBean seems to be not serializable, so you need to convert to
>> other data type.
>>
>> Hope this helps.
>>
>> Regards.
>> Jungtaek Lim (HeartSaVioR)
>>
>>
>>
>> 2015-05-14 18:38 GMT+09:00 Asif Ihsan <asifihsan.ihsan@gmail.com
>> <javascript:_e(%7B%7D,'cvml','asifihsan.ihsan@gmail.com');>>:
>>
>>> I am using Esper in one of the bolts of the Storm topology. Bolt emit
>>> MapEventBean array (EventBean[]). With the single worker topology run
>>> smoothly. With multiple workers it gives following error in emit function
>>> call.
>>>
>>> May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
>>> SEVERE: Async loop died!
>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>> java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
>>> at
>>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
>>> at
>>> backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
>>> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
>>> at clojure.lang.AFn.run(AFn.java:24)
>>> at java.lang.Thread.run(Unknown Source)
>>> Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
>>> com.espertech.esper.event.map.MapEventBean
>>> at
>>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
>>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>>> at
>>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
>>> at
>>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
>>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>>> at
>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
>>> at
>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
>>> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
>>> at
>>> backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
>>> at
>>> backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
>>> at
>>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
>>> at backtype.storm.util$fast_list_map.invoke(util.clj:771)
>>> at
>>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
>>> at
>>> backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
>>> at
>>> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
>>> ... 6 more
>>> Caused by: java.io.NotSerializableException:
>>> com.espertech.esper.event.map.MapEventBean
>>> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>>> at java.io.ObjectOutputStream.writeArray(Unknown Source)
>>> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>>> at java.io.ObjectOutputStream.writeObject(Unknown Source)
>>> at
>>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
>>> ... 21 more
>>>
>>> May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
>>> SEVERE:
>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>> java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
>>> at
>>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
>>> at
>>> backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
>>> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
>>> at clojure.lang.AFn.run(AFn.java:24)
>>> at java.lang.Thread.run(Unknown Source)
>>> Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
>>> com.espertech.esper.event.map.MapEventBean
>>> at
>>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
>>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>>> at
>>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
>>> at
>>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
>>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>>> at
>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
>>> at
>>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
>>> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
>>> at
>>> backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
>>> at
>>> backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
>>> at
>>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
>>> at backtype.storm.util$fast_list_map.invoke(util.clj:771)
>>> at
>>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
>>> at
>>> backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
>>> at
>>> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
>>> ... 6 more
>>> Caused by: java.io.NotSerializableException:
>>> com.espertech.esper.event.map.MapEventBean
>>> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>>> at java.io.ObjectOutputStream.writeArray(Unknown Source)
>>> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>>> at java.io.ObjectOutputStream.writeObject(Unknown Source)
>>> at
>>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
>>> ... 21 more
>>>
>>>
>>> On Wed, May 13, 2015 at 6:11 PM, Jeffery Maass <maassql@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','maassql@gmail.com');>> wrote:
>>>
>>>> Asif:
>>>>
>>>> Would love to help you, however, we will need way more details. Why
>>>> don't you start with telling us what the error is? How are you detecting
>>>> the error?
>>>>
>>>> You will want to look over both the supervisor and worker logs.
>>>>
>>>> Thank you for your time!
>>>>
>>>> +++++++++++++++++++++
>>>> Jeff Maass <javascript:_e(%7B%7D,'cvml','maassql@gmail.com');>
>>>> linkedin.com/in/jeffmaass
>>>> stackoverflow.com/users/373418/maassql
>>>> +++++++++++++++++++++
>>>>
>>>>
>>>> On Wed, May 13, 2015 at 2:37 AM, Asif Ihsan <asifihsan.ihsan@gmail.com
>>>> <javascript:_e(%7B%7D,'cvml','asifihsan.ihsan@gmail.com');>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I recently started using Storm. I am having topology. It works well
>>>>> with numofworkers=1. However when I change it to 6 or more than 1. Its
>>>>> gives error when emitting tuples. What is the issue and how to resolve it
>>>>> with multiple workers. Thank you.
>>>>>
>>>>> --
>>>>> Regards
>>>>> Muhammad Asif Ihsan
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards
>>> Muhammad Asif Ihsan
>>>
>>
>>
>>
>> --
>> Name : 임 정택
>> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
>> Twitter : http://twitter.com/heartsavior
>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>
>
>
>
> --
> Regards
> Muhammad Asif Ihsan
>
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
Re: Multiple Workers in local mode gives error
Posted by Asif Ihsan <as...@gmail.com>.
I am using shuffelgrouping. With one worker it is running well.
On Thu, May 14, 2015 at 4:18 PM, <Ra...@dellteam.com> wrote:
> *Dell - Internal Use - Confidential *
>
> Regarding the NotSerializableException:
>
>
>
> I would like to see what grouping you have defined between the two bolts
> that you have experienced the NotSerializableException with.
>
>
>
> I am guessing it is localOrShuffleGrouping, can you change it to
> shuffleGrouping and retest with one worker.
>
>
>
> *From:* Asif Ihsan [mailto:asifihsan.ihsan@gmail.com]
> *Sent:* Thursday, May 14, 2015 5:23 AM
> *To:* user@storm.apache.org
> *Subject:* Re: Multiple Workers in local mode gives error
>
>
>
> You are right about it. What does multiple workers means. Does it mean
> that when I run topology with single worker than i will see one single
> process handling all the bolts and when running 6 workers for 5 bolts and 1
> spout than i will see 6 separate processes running each bolt and spout? M i
> right.If not then what does it mean?
>
>
>
>
>
> On Thu, May 14, 2015 at 2:46 PM, 임정택 <ka...@gmail.com> wrote:
>
> Hi.
>
>
>
> Storm serializes tuples when tuple should be sent to other (remote) worker.
>
> In other words, Storm doesn't serialize tuples when destination is local
> task. That's why you didn't meet error when testing with 1 worker.
>
>
>
> MapEventBean seems to be not serializable, so you need to convert to other
> data type.
>
>
>
> Hope this helps.
>
>
>
> Regards.
>
> Jungtaek Lim (HeartSaVioR)
>
>
>
>
>
>
>
> 2015-05-14 18:38 GMT+09:00 Asif Ihsan <as...@gmail.com>:
>
> I am using Esper in one of the bolts of the Storm topology. Bolt emit
> MapEventBean array (EventBean[]). With the single worker topology run
> smoothly. With multiple workers it gives following error in emit function
> call.
>
>
>
> May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
>
> SEVERE: Async loop died!
>
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
>
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
>
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
>
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
>
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
>
> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
>
> at clojure.lang.AFn.run(AFn.java:24)
>
> at java.lang.Thread.run(Unknown Source)
>
> Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
> com.espertech.esper.event.map.MapEventBean
>
> at
> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
>
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
>
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
>
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
>
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
>
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
>
> at
> backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
>
> at
> backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
>
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
>
> at backtype.storm.util$fast_list_map.invoke(util.clj:771)
>
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
>
> at
> backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
>
> at
> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
>
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
>
> ... 6 more
>
> Caused by: java.io.NotSerializableException:
> com.espertech.esper.event.map.MapEventBean
>
> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>
> at java.io.ObjectOutputStream.writeArray(Unknown Source)
>
> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>
> at java.io.ObjectOutputStream.writeObject(Unknown Source)
>
> at
> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
>
> ... 21 more
>
>
>
> May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
>
> SEVERE:
>
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
>
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
>
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
>
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
>
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
>
> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
>
> at clojure.lang.AFn.run(AFn.java:24)
>
> at java.lang.Thread.run(Unknown Source)
>
> Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
> com.espertech.esper.event.map.MapEventBean
>
> at
> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
>
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
>
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
>
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
>
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
>
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
>
> at
> backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
>
> at
> backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
>
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
>
> at backtype.storm.util$fast_list_map.invoke(util.clj:771)
>
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
>
> at
> backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
>
> at
> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
>
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
>
> ... 6 more
>
> Caused by: java.io.NotSerializableException:
> com.espertech.esper.event.map.MapEventBean
>
> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>
> at java.io.ObjectOutputStream.writeArray(Unknown Source)
>
> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>
> at java.io.ObjectOutputStream.writeObject(Unknown Source)
>
> at
> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
>
> ... 21 more
>
>
>
>
>
> On Wed, May 13, 2015 at 6:11 PM, Jeffery Maass <ma...@gmail.com> wrote:
>
> Asif:
>
> Would love to help you, however, we will need way more details. Why don't
> you start with telling us what the error is? How are you detecting the
> error?
>
> You will want to look over both the supervisor and worker logs.
>
>
> Thank you for your time!
>
> +++++++++++++++++++++
> Jeff Maass <ma...@gmail.com>
> linkedin.com/in/jeffmaass
> stackoverflow.com/users/373418/maassql
> +++++++++++++++++++++
>
>
>
> On Wed, May 13, 2015 at 2:37 AM, Asif Ihsan <as...@gmail.com>
> wrote:
>
> Hi,
>
>
>
> I recently started using Storm. I am having topology. It works well with
> numofworkers=1. However when I change it to 6 or more than 1. Its gives
> error when emitting tuples. What is the issue and how to resolve it with
> multiple workers. Thank you.
>
>
>
> --
>
> Regards
>
> Muhammad Asif Ihsan
>
>
>
>
>
>
>
> --
>
> Regards
>
> Muhammad Asif Ihsan
>
>
>
>
>
> --
>
> Name : 임 정택
> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
> Twitter : http://twitter.com/heartsavior
>
> LinkedIn : http://www.linkedin.com/in/heartsavior
>
>
>
>
>
> --
>
> Regards
>
> Muhammad Asif Ihsan
>
--
Regards
Muhammad Asif Ihsan
RE: Multiple Workers in local mode gives error
Posted by Ra...@DellTeam.com.
Dell - Internal Use - Confidential
Regarding the NotSerializableException:
I would like to see what grouping you have defined between the two bolts that you have experienced the NotSerializableException with.
I am guessing it is localOrShuffleGrouping, can you change it to shuffleGrouping and retest with one worker.
From: Asif Ihsan [mailto:asifihsan.ihsan@gmail.com]
Sent: Thursday, May 14, 2015 5:23 AM
To: user@storm.apache.org
Subject: Re: Multiple Workers in local mode gives error
You are right about it. What does multiple workers means. Does it mean that when I run topology with single worker than i will see one single process handling all the bolts and when running 6 workers for 5 bolts and 1 spout than i will see 6 separate processes running each bolt and spout? M i right.If not then what does it mean?
On Thu, May 14, 2015 at 2:46 PM, 임정택 <ka...@gmail.com>> wrote:
Hi.
Storm serializes tuples when tuple should be sent to other (remote) worker.
In other words, Storm doesn't serialize tuples when destination is local task. That's why you didn't meet error when testing with 1 worker.
MapEventBean seems to be not serializable, so you need to convert to other data type.
Hope this helps.
Regards.
Jungtaek Lim (HeartSaVioR)
2015-05-14 18:38 GMT+09:00 Asif Ihsan <as...@gmail.com>>:
I am using Esper in one of the bolts of the Storm topology. Bolt emit MapEventBean array (EventBean[]). With the single worker topology run smoothly. With multiple workers it gives following error in emit function call.
May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
SEVERE: Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
at backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
at backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
at backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
at backtype.storm.util$fast_list_map.invoke(util.clj:771)
at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
at backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
... 6 more
Caused by: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeArray(Unknown Source)
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeObject(Unknown Source)
at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
... 21 more
May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
SEVERE:
java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
at backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
at backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
at backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
at backtype.storm.util$fast_list_map.invoke(util.clj:771)
at backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
at backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
at backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
... 6 more
Caused by: java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeArray(Unknown Source)
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeObject(Unknown Source)
at backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
... 21 more
On Wed, May 13, 2015 at 6:11 PM, Jeffery Maass <ma...@gmail.com>> wrote:
Asif:
Would love to help you, however, we will need way more details. Why don't you start with telling us what the error is? How are you detecting the error?
You will want to look over both the supervisor and worker logs.
Thank you for your time!
+++++++++++++++++++++
Jeff Maass<ma...@gmail.com>
linkedin.com/in/jeffmaass<http://linkedin.com/in/jeffmaass>
stackoverflow.com/users/373418/maassql<http://stackoverflow.com/users/373418/maassql>
+++++++++++++++++++++
On Wed, May 13, 2015 at 2:37 AM, Asif Ihsan <as...@gmail.com>> wrote:
Hi,
I recently started using Storm. I am having topology. It works well with numofworkers=1. However when I change it to 6 or more than 1. Its gives error when emitting tuples. What is the issue and how to resolve it with multiple workers. Thank you.
--
Regards
Muhammad Asif Ihsan
--
Regards
Muhammad Asif Ihsan
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
--
Regards
Muhammad Asif Ihsan
Re: Multiple Workers in local mode gives error
Posted by Asif Ihsan <as...@gmail.com>.
You are right about it. What does multiple workers means. Does it mean that
when I run topology with single worker than i will see one single process
handling all the bolts and when running 6 workers for 5 bolts and 1 spout
than i will see 6 separate processes running each bolt and spout? M i
right.If not then what does it mean?
On Thu, May 14, 2015 at 2:46 PM, 임정택 <ka...@gmail.com> wrote:
> Hi.
>
> Storm serializes tuples when tuple should be sent to other (remote) worker.
> In other words, Storm doesn't serialize tuples when destination is local
> task. That's why you didn't meet error when testing with 1 worker.
>
> MapEventBean seems to be not serializable, so you need to convert to other
> data type.
>
> Hope this helps.
>
> Regards.
> Jungtaek Lim (HeartSaVioR)
>
>
>
> 2015-05-14 18:38 GMT+09:00 Asif Ihsan <as...@gmail.com>:
>
>> I am using Esper in one of the bolts of the Storm topology. Bolt emit
>> MapEventBean array (EventBean[]). With the single worker topology run
>> smoothly. With multiple workers it gives following error in emit function
>> call.
>>
>> May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
>> SEVERE: Async loop died!
>> java.lang.RuntimeException: java.lang.RuntimeException:
>> java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
>> at
>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
>> at
>> backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
>> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
>> at clojure.lang.AFn.run(AFn.java:24)
>> at java.lang.Thread.run(Unknown Source)
>> Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
>> com.espertech.esper.event.map.MapEventBean
>> at
>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>> at
>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
>> at
>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>> at
>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
>> at
>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
>> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
>> at
>> backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
>> at
>> backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
>> at
>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
>> at backtype.storm.util$fast_list_map.invoke(util.clj:771)
>> at
>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
>> at
>> backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
>> at
>> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
>> ... 6 more
>> Caused by: java.io.NotSerializableException:
>> com.espertech.esper.event.map.MapEventBean
>> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>> at java.io.ObjectOutputStream.writeArray(Unknown Source)
>> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>> at java.io.ObjectOutputStream.writeObject(Unknown Source)
>> at
>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
>> ... 21 more
>>
>> May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
>> SEVERE:
>> java.lang.RuntimeException: java.lang.RuntimeException:
>> java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
>> at
>> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
>> at
>> backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
>> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
>> at clojure.lang.AFn.run(AFn.java:24)
>> at java.lang.Thread.run(Unknown Source)
>> Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
>> com.espertech.esper.event.map.MapEventBean
>> at
>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>> at
>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
>> at
>> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
>> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
>> at
>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
>> at
>> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
>> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
>> at
>> backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
>> at
>> backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
>> at
>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
>> at backtype.storm.util$fast_list_map.invoke(util.clj:771)
>> at
>> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
>> at
>> backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
>> at
>> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
>> ... 6 more
>> Caused by: java.io.NotSerializableException:
>> com.espertech.esper.event.map.MapEventBean
>> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>> at java.io.ObjectOutputStream.writeArray(Unknown Source)
>> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
>> at java.io.ObjectOutputStream.writeObject(Unknown Source)
>> at
>> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
>> ... 21 more
>>
>>
>> On Wed, May 13, 2015 at 6:11 PM, Jeffery Maass <ma...@gmail.com> wrote:
>>
>>> Asif:
>>>
>>> Would love to help you, however, we will need way more details. Why
>>> don't you start with telling us what the error is? How are you detecting
>>> the error?
>>>
>>> You will want to look over both the supervisor and worker logs.
>>>
>>> Thank you for your time!
>>>
>>> +++++++++++++++++++++
>>> Jeff Maass <ma...@gmail.com>
>>> linkedin.com/in/jeffmaass
>>> stackoverflow.com/users/373418/maassql
>>> +++++++++++++++++++++
>>>
>>>
>>> On Wed, May 13, 2015 at 2:37 AM, Asif Ihsan <as...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I recently started using Storm. I am having topology. It works well
>>>> with numofworkers=1. However when I change it to 6 or more than 1. Its
>>>> gives error when emitting tuples. What is the issue and how to resolve it
>>>> with multiple workers. Thank you.
>>>>
>>>> --
>>>> Regards
>>>> Muhammad Asif Ihsan
>>>>
>>>
>>>
>>
>>
>> --
>> Regards
>> Muhammad Asif Ihsan
>>
>
>
>
> --
> Name : 임 정택
> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>
--
Regards
Muhammad Asif Ihsan
Re: Multiple Workers in local mode gives error
Posted by 임정택 <ka...@gmail.com>.
Hi.
Storm serializes tuples when tuple should be sent to other (remote) worker.
In other words, Storm doesn't serialize tuples when destination is local
task. That's why you didn't meet error when testing with 1 worker.
MapEventBean seems to be not serializable, so you need to convert to other
data type.
Hope this helps.
Regards.
Jungtaek Lim (HeartSaVioR)
2015-05-14 18:38 GMT+09:00 Asif Ihsan <as...@gmail.com>:
> I am using Esper in one of the bolts of the Storm topology. Bolt emit
> MapEventBean array (EventBean[]). With the single worker topology run
> smoothly. With multiple workers it gives following error in emit function
> call.
>
> May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
> SEVERE: Async loop died!
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
> com.espertech.esper.event.map.MapEventBean
> at
> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
> at
> backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
> at
> backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
> at backtype.storm.util$fast_list_map.invoke(util.clj:771)
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
> at
> backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
> at
> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
> ... 6 more
> Caused by: java.io.NotSerializableException:
> com.espertech.esper.event.map.MapEventBean
> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
> at java.io.ObjectOutputStream.writeArray(Unknown Source)
> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
> at java.io.ObjectOutputStream.writeObject(Unknown Source)
> at
> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
> ... 21 more
>
> May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
> SEVERE:
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
> at
> backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
> com.espertech.esper.event.map.MapEventBean
> at
> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
> at
> backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
> at
> backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
> at backtype.storm.util$fast_list_map.invoke(util.clj:771)
> at
> backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
> at
> backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
> at
> backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
> ... 6 more
> Caused by: java.io.NotSerializableException:
> com.espertech.esper.event.map.MapEventBean
> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
> at java.io.ObjectOutputStream.writeArray(Unknown Source)
> at java.io.ObjectOutputStream.writeObject0(Unknown Source)
> at java.io.ObjectOutputStream.writeObject(Unknown Source)
> at
> backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
> ... 21 more
>
>
> On Wed, May 13, 2015 at 6:11 PM, Jeffery Maass <ma...@gmail.com> wrote:
>
>> Asif:
>>
>> Would love to help you, however, we will need way more details. Why
>> don't you start with telling us what the error is? How are you detecting
>> the error?
>>
>> You will want to look over both the supervisor and worker logs.
>>
>> Thank you for your time!
>>
>> +++++++++++++++++++++
>> Jeff Maass <ma...@gmail.com>
>> linkedin.com/in/jeffmaass
>> stackoverflow.com/users/373418/maassql
>> +++++++++++++++++++++
>>
>>
>> On Wed, May 13, 2015 at 2:37 AM, Asif Ihsan <as...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I recently started using Storm. I am having topology. It works well with
>>> numofworkers=1. However when I change it to 6 or more than 1. Its gives
>>> error when emitting tuples. What is the issue and how to resolve it with
>>> multiple workers. Thank you.
>>>
>>> --
>>> Regards
>>> Muhammad Asif Ihsan
>>>
>>
>>
>
>
> --
> Regards
> Muhammad Asif Ihsan
>
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
Re: Multiple Workers in local mode gives error
Posted by Asif Ihsan <as...@gmail.com>.
I am using Esper in one of the bolts of the Storm topology. Bolt emit
MapEventBean array (EventBean[]). With the single worker topology run
smoothly. With multiple workers it gives following error in emit function
call.
May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
SEVERE: Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException:
java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
com.espertech.esper.event.map.MapEventBean
at
backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at
com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
at
com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
at
backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
at
backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
at
backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
at backtype.storm.util$fast_list_map.invoke(util.clj:771)
at
backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
at
backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
at
backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
... 6 more
Caused by: java.io.NotSerializableException:
com.espertech.esper.event.map.MapEventBean
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeArray(Unknown Source)
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeObject(Unknown Source)
at
backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
... 21 more
May 14, 2015 2:37:07 PM clojure.tools.logging$eval1$fn__7 invoke
SEVERE:
java.lang.RuntimeException: java.lang.RuntimeException:
java.io.NotSerializableException: com.espertech.esper.event.map.MapEventBean
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:87)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:58)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:62)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1619.invoke(disruptor.clj:73)
at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: java.io.NotSerializableException:
com.espertech.esper.event.map.MapEventBean
at
backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:24)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at
com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:87)
at
com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:17)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:554)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:77)
at
com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:18)
at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:472)
at
backtype.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:27)
at
backtype.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:27)
at
backtype.storm.daemon.worker$mk_transfer_fn$fn__4152$fn__4156.invoke(worker.clj:99)
at backtype.storm.util$fast_list_map.invoke(util.clj:771)
at
backtype.storm.daemon.worker$mk_transfer_fn$fn__4152.invoke(worker.clj:99)
at
backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3914.invoke(executor.clj:238)
at
backtype.storm.disruptor$clojure_handler$reify__1606.onEvent(disruptor.clj:43)
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:84)
... 6 more
Caused by: java.io.NotSerializableException:
com.espertech.esper.event.map.MapEventBean
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeArray(Unknown Source)
at java.io.ObjectOutputStream.writeObject0(Unknown Source)
at java.io.ObjectOutputStream.writeObject(Unknown Source)
at
backtype.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:21)
... 21 more
On Wed, May 13, 2015 at 6:11 PM, Jeffery Maass <ma...@gmail.com> wrote:
> Asif:
>
> Would love to help you, however, we will need way more details. Why don't
> you start with telling us what the error is? How are you detecting the
> error?
>
> You will want to look over both the supervisor and worker logs.
>
> Thank you for your time!
>
> +++++++++++++++++++++
> Jeff Maass <ma...@gmail.com>
> linkedin.com/in/jeffmaass
> stackoverflow.com/users/373418/maassql
> +++++++++++++++++++++
>
>
> On Wed, May 13, 2015 at 2:37 AM, Asif Ihsan <as...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I recently started using Storm. I am having topology. It works well with
>> numofworkers=1. However when I change it to 6 or more than 1. Its gives
>> error when emitting tuples. What is the issue and how to resolve it with
>> multiple workers. Thank you.
>>
>> --
>> Regards
>> Muhammad Asif Ihsan
>>
>
>
--
Regards
Muhammad Asif Ihsan
Re: Multiple Workers in local mode gives error
Posted by Jeffery Maass <ma...@gmail.com>.
Asif:
Would love to help you, however, we will need way more details. Why don't
you start with telling us what the error is? How are you detecting the
error?
You will want to look over both the supervisor and worker logs.
Thank you for your time!
+++++++++++++++++++++
Jeff Maass <ma...@gmail.com>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++
On Wed, May 13, 2015 at 2:37 AM, Asif Ihsan <as...@gmail.com>
wrote:
> Hi,
>
> I recently started using Storm. I am having topology. It works well with
> numofworkers=1. However when I change it to 6 or more than 1. Its gives
> error when emitting tuples. What is the issue and how to resolve it with
> multiple workers. Thank you.
>
> --
> Regards
> Muhammad Asif Ihsan
>