You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Jie Li <ja...@gmail.com> on 2014/08/13 18:47:59 UTC
transient errors of "Tuple created with wrong number of fields"
Hi all,
We've been testing Storm for a while and have been happy so far, but this
error seems a bit unknown and concerning. One of our topologies somehow
came across this error during testing with nothing really changed, so I
have no clue how to debug it. And it seems an transient error, as the
topology somehow recovered itself after running for a while.
The storm version is 0.9.1 running with most of default settings. One
thing I noticed suspicious was the warning message of "Received invalid
messages for unknown tasks. Dropping... ". Not sure if it was related and
if anyone has idea? Here is the full stack trace of the error (that doesn't
seem useful):
java.lang.RuntimeException: java.lang.IllegalArgumentException: Tuple
created with wrong number of fields. Expected 2 fields but got 1 fields
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:66)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
at backtype.storm.disruptor$consume_batch.invoke(disruptor.clj:74)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
at
backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:539)
~[na:na]
at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
~[na:na]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38]
Caused by: java.lang.IllegalArgumentException: Tuple created with wrong
number of fields. Expected 2 fields but got 1 fields
at backtype.storm.tuple.TupleImpl.<init>(TupleImpl.java:55)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
at
backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:54)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__3839.invoke(executor.clj:395)
~[na:na]
at
backtype.storm.disruptor$clojure_handler$reify__1560.onEvent(disruptor.clj:58)
~[na:na]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:104)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
... 6 common frames omitted
2014-08-13 01:14:58 b.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.IllegalArgumentException: Tuple
created with wrong number of fields. Expected 2 fields but got 1 fields
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:66)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
at backtype.storm.disruptor$consume_batch.invoke(disruptor.clj:74)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
at
backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:539)
~[na:na]
at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
~[na:na]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38]
Caused by: java.lang.IllegalArgumentException: Tuple created with wrong
number of fields. Expected 2 fields but got 1 fields
at backtype.storm.tuple.TupleImpl.<init>(TupleImpl.java:55)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
at
backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:54)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__3839.invoke(executor.clj:395)
~[na:na]
at
backtype.storm.disruptor$clojure_handler$reify__1560.onEvent(disruptor.clj:58)
~[na:na]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:104)
~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
... 6 common frames omitted
2014-08-13 01:14:58 b.s.util [INFO] Halting process: ("Worker died")
Thanks,
Jie Li
Software engineer at Pinterest | www.linkedin.com/in/jay23jack/
Re: transient errors of "Tuple created with wrong number of fields"
Posted by Nathan Marz <na...@nathanmarz.com>.
Hi Jie,
A few questions:
1. How many topologies do you have running on this cluster?
2. Once the topology recovers, do the errors still happen or do they
disappear for the most part?
3. Have you noticed any worker failures on the cluster around the time
these errors happen?
-Nathan
On Tue, Aug 26, 2014 at 10:04 PM, Jie Li <ja...@gmail.com> wrote:
> Thanks Dmitry. We are using Trident with transactional spout/state, and
> this happens in the first 10 minutes when the topolgoy starts, and recover
> itself after then. Since the topology is transactional, same messages are
> replayed, so this doesn't seem a data issue, and does seem some storm bug?
>
> Jie Li
> Software engineer at Pinterest | www.linkedin.com/in/jay23jack/
>
>
> On Wed, Aug 13, 2014 at 2:13 PM, Dmitry Shulga <dm...@gmail.com>
> wrote:
>
>> If you have access to the bolt, check what fields the bolt is expecting.
>> You may have some coding issues when creating a Tuple for it.
>>
>>
>>
>>
>> Here is example of the method that defines what fields the bolt is outputting taken from https://storm.incubator.apache.org/documentation/Tutorial.html. The Execute method emits a Tuple with two fields, definitions of which can be found in declareOutputFields.
>>
>>
>>
>> @Override
>> public void execute(Tuple input) {
>> int val = input.getInteger(0);
>> _collector.emit(input, new Values(val*2, val*3));
>> _collector.ack(input);
>> }
>>
>> @Override
>> public void declareOutputFields(OutputFieldsDeclarer declarer) {
>> declarer.declare(new Fields("double", "triple"));
>> }
>>
>>
>>
>> -D
>>
>>
>> On Wed, Aug 13, 2014 at 9:47 AM, Jie Li <ja...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> We've been testing Storm for a while and have been happy so far, but
>>> this error seems a bit unknown and concerning. One of our topologies
>>> somehow came across this error during testing with nothing really changed,
>>> so I have no clue how to debug it. And it seems an transient error, as the
>>> topology somehow recovered itself after running for a while.
>>>
>>> The storm version is 0.9.1 running with most of default settings. One
>>> thing I noticed suspicious was the warning message of "Received invalid
>>> messages for unknown tasks. Dropping... ". Not sure if it was related and
>>> if anyone has idea? Here is the full stack trace of the error (that doesn't
>>> seem useful):
>>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Tuple
>>> created with wrong number of fields. Expected 2 fields but got 1 fields
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
>>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:66)
>>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>>> at
>>> backtype.storm.disruptor$consume_batch.invoke(disruptor.clj:74)
>>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>>> at
>>> backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:539)
>>> ~[na:na]
>>> at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
>>> ~[na:na]
>>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
>>> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38]
>>> Caused by: java.lang.IllegalArgumentException: Tuple created with wrong
>>> number of fields. Expected 2 fields but got 1 fields
>>> at backtype.storm.tuple.TupleImpl.<init>(TupleImpl.java:55)
>>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>>> at
>>> backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:54)
>>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>>> at
>>> backtype.storm.daemon.executor$mk_task_receiver$fn__3839.invoke(executor.clj:395)
>>> ~[na:na]
>>> at
>>> backtype.storm.disruptor$clojure_handler$reify__1560.onEvent(disruptor.clj:58)
>>> ~[na:na]
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:104)
>>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>>> ... 6 common frames omitted
>>> 2014-08-13 01:14:58 b.s.d.executor [ERROR]
>>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Tuple
>>> created with wrong number of fields. Expected 2 fields but got 1 fields
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
>>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:66)
>>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>>> at
>>> backtype.storm.disruptor$consume_batch.invoke(disruptor.clj:74)
>>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>>> at
>>> backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:539)
>>> ~[na:na]
>>> at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
>>> ~[na:na]
>>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
>>> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38]
>>> Caused by: java.lang.IllegalArgumentException: Tuple created with wrong
>>> number of fields. Expected 2 fields but got 1 fields
>>> at backtype.storm.tuple.TupleImpl.<init>(TupleImpl.java:55)
>>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>>> at
>>> backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:54)
>>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>>> at
>>> backtype.storm.daemon.executor$mk_task_receiver$fn__3839.invoke(executor.clj:395)
>>> ~[na:na]
>>> at
>>> backtype.storm.disruptor$clojure_handler$reify__1560.onEvent(disruptor.clj:58)
>>> ~[na:na]
>>> at
>>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:104)
>>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>>> ... 6 common frames omitted
>>> 2014-08-13 01:14:58 b.s.util [INFO] Halting process: ("Worker died")
>>> Thanks,
>>> Jie Li
>>> Software engineer at Pinterest | www.linkedin.com/in/jay23jack/
>>>
>>
>>
>
--
Twitter: @nathanmarz
http://nathanmarz.com
Re: transient errors of "Tuple created with wrong number of fields"
Posted by Jie Li <ja...@gmail.com>.
Thanks Dmitry. We are using Trident with transactional spout/state, and
this happens in the first 10 minutes when the topolgoy starts, and recover
itself after then. Since the topology is transactional, same messages are
replayed, so this doesn't seem a data issue, and does seem some storm bug?
Jie Li
Software engineer at Pinterest | www.linkedin.com/in/jay23jack/
On Wed, Aug 13, 2014 at 2:13 PM, Dmitry Shulga <dm...@gmail.com>
wrote:
> If you have access to the bolt, check what fields the bolt is expecting.
> You may have some coding issues when creating a Tuple for it.
>
>
>
> Here is example of the method that defines what fields the bolt is outputting taken from https://storm.incubator.apache.org/documentation/Tutorial.html. The Execute method emits a Tuple with two fields, definitions of which can be found in declareOutputFields.
>
>
> @Override
> public void execute(Tuple input) {
> int val = input.getInteger(0);
> _collector.emit(input, new Values(val*2, val*3));
> _collector.ack(input);
> }
>
> @Override
> public void declareOutputFields(OutputFieldsDeclarer declarer) {
> declarer.declare(new Fields("double", "triple"));
> }
>
>
>
> -D
>
>
> On Wed, Aug 13, 2014 at 9:47 AM, Jie Li <ja...@gmail.com> wrote:
>
>> Hi all,
>>
>> We've been testing Storm for a while and have been happy so far, but this
>> error seems a bit unknown and concerning. One of our topologies somehow
>> came across this error during testing with nothing really changed, so I
>> have no clue how to debug it. And it seems an transient error, as the
>> topology somehow recovered itself after running for a while.
>>
>> The storm version is 0.9.1 running with most of default settings. One
>> thing I noticed suspicious was the warning message of "Received invalid
>> messages for unknown tasks. Dropping... ". Not sure if it was related and
>> if anyone has idea? Here is the full stack trace of the error (that doesn't
>> seem useful):
>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Tuple
>> created with wrong number of fields. Expected 2 fields but got 1 fields
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:66)
>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>> at
>> backtype.storm.disruptor$consume_batch.invoke(disruptor.clj:74)
>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>> at
>> backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:539)
>> ~[na:na]
>> at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
>> ~[na:na]
>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
>> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38]
>> Caused by: java.lang.IllegalArgumentException: Tuple created with wrong
>> number of fields. Expected 2 fields but got 1 fields
>> at backtype.storm.tuple.TupleImpl.<init>(TupleImpl.java:55)
>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>> at
>> backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:54)
>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>> at
>> backtype.storm.daemon.executor$mk_task_receiver$fn__3839.invoke(executor.clj:395)
>> ~[na:na]
>> at
>> backtype.storm.disruptor$clojure_handler$reify__1560.onEvent(disruptor.clj:58)
>> ~[na:na]
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:104)
>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>> ... 6 common frames omitted
>> 2014-08-13 01:14:58 b.s.d.executor [ERROR]
>> java.lang.RuntimeException: java.lang.IllegalArgumentException: Tuple
>> created with wrong number of fields. Expected 2 fields but got 1 fields
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:66)
>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>> at
>> backtype.storm.disruptor$consume_batch.invoke(disruptor.clj:74)
>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>> at
>> backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:539)
>> ~[na:na]
>> at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
>> ~[na:na]
>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
>> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38]
>> Caused by: java.lang.IllegalArgumentException: Tuple created with wrong
>> number of fields. Expected 2 fields but got 1 fields
>> at backtype.storm.tuple.TupleImpl.<init>(TupleImpl.java:55)
>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>> at
>> backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:54)
>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>> at
>> backtype.storm.daemon.executor$mk_task_receiver$fn__3839.invoke(executor.clj:395)
>> ~[na:na]
>> at
>> backtype.storm.disruptor$clojure_handler$reify__1560.onEvent(disruptor.clj:58)
>> ~[na:na]
>> at
>> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:104)
>> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
>> ... 6 common frames omitted
>> 2014-08-13 01:14:58 b.s.util [INFO] Halting process: ("Worker died")
>> Thanks,
>> Jie Li
>> Software engineer at Pinterest | www.linkedin.com/in/jay23jack/
>>
>
>
Re: transient errors of "Tuple created with wrong number of fields"
Posted by Dmitry Shulga <dm...@gmail.com>.
If you have access to the bolt, check what fields the bolt is expecting.
You may have some coding issues when creating a Tuple for it.
Here is example of the method that defines what fields the bolt is
outputting taken from
https://storm.incubator.apache.org/documentation/Tutorial.html. The
Execute method emits a Tuple with two fields, definitions of which can
be found in declareOutputFields.
@Override
public void execute(Tuple input) {
int val = input.getInteger(0);
_collector.emit(input, new Values(val*2, val*3));
_collector.ack(input);
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("double", "triple"));
}
-D
On Wed, Aug 13, 2014 at 9:47 AM, Jie Li <ja...@gmail.com> wrote:
> Hi all,
>
> We've been testing Storm for a while and have been happy so far, but this
> error seems a bit unknown and concerning. One of our topologies somehow
> came across this error during testing with nothing really changed, so I
> have no clue how to debug it. And it seems an transient error, as the
> topology somehow recovered itself after running for a while.
>
> The storm version is 0.9.1 running with most of default settings. One
> thing I noticed suspicious was the warning message of "Received invalid
> messages for unknown tasks. Dropping... ". Not sure if it was related and
> if anyone has idea? Here is the full stack trace of the error (that doesn't
> seem useful):
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Tuple
> created with wrong number of fields. Expected 2 fields but got 1 fields
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:66)
> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
> at backtype.storm.disruptor$consume_batch.invoke(disruptor.clj:74)
> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
> at
> backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:539)
> ~[na:na]
> at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
> ~[na:na]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38]
> Caused by: java.lang.IllegalArgumentException: Tuple created with wrong
> number of fields. Expected 2 fields but got 1 fields
> at backtype.storm.tuple.TupleImpl.<init>(TupleImpl.java:55)
> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
> at
> backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:54)
> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__3839.invoke(executor.clj:395)
> ~[na:na]
> at
> backtype.storm.disruptor$clojure_handler$reify__1560.onEvent(disruptor.clj:58)
> ~[na:na]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:104)
> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
> ... 6 common frames omitted
> 2014-08-13 01:14:58 b.s.d.executor [ERROR]
> java.lang.RuntimeException: java.lang.IllegalArgumentException: Tuple
> created with wrong number of fields. Expected 2 fields but got 1 fields
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:66)
> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
> at backtype.storm.disruptor$consume_batch.invoke(disruptor.clj:74)
> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
> at
> backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:539)
> ~[na:na]
> at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
> ~[na:na]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_38]
> Caused by: java.lang.IllegalArgumentException: Tuple created with wrong
> number of fields. Expected 2 fields but got 1 fields
> at backtype.storm.tuple.TupleImpl.<init>(TupleImpl.java:55)
> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
> at
> backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:54)
> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__3839.invoke(executor.clj:395)
> ~[na:na]
> at
> backtype.storm.disruptor$clojure_handler$reify__1560.onEvent(disruptor.clj:58)
> ~[na:na]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:104)
> ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating]
> ... 6 common frames omitted
> 2014-08-13 01:14:58 b.s.util [INFO] Halting process: ("Worker died")
> Thanks,
> Jie Li
> Software engineer at Pinterest | www.linkedin.com/in/jay23jack/
>