Query about failure conditions in Storm

[Calvin Iyer <ca...@gmail.com>]

Hi,

I have a query about what happens if a storm bolt dies. My current topology
is such that I have a Kafka spout that gives out records which are counted
in a storm bolt and stored in a hashmap in the same bolt and emits the
aggregated map to another bolt every 5 mins to persist it to a database.
Now supposing that the bolt dies while having data in the map and the task
gets reassigned to another worker does the map also get recreated or does
it use the last snapshot of the map before it failed ?

Thanks,
Calvin

Re: Query about failure conditions in Storm

[Calvin Iyer <ca...@gmail.com>]

Thanks a lot Martin that clears things up.

On Mon, Sep 7, 2015 at 2:47 PM, Martin Burian <ma...@gmail.com>
wrote:

> Hi,
> the bolt is created and prepared the same way as when the topology is
> created. You need to recover your map from the DB yourself, in the
> prepare() method of your bolt. Also note that the spout will keep emitting
> tuples even when the bolt is dead, so you had better set the
> topology.max.pending property to keep the buffers from filling up and the
> tuple timeout to accomodate the time your bolt takes to recover.
> Hope it helps,
> Martin
>
> po 7. 9. 2015 v 8:59 odesílatel Calvin Iyer <ca...@gmail.com>
> napsal:
>
>> Hi,
>>
>> I have a query about what happens if a storm bolt dies. My current
>> topology is such that I have a Kafka spout that gives out records which are
>> counted in a storm bolt and stored in a hashmap in the same bolt and emits
>> the aggregated map to another bolt every 5 mins to persist it to a
>> database. Now supposing that the bolt dies while having data in the map and
>> the task gets reassigned to another worker does the map also get recreated
>> or does it use the last snapshot of the map before it failed ?
>>
>> Thanks,
>> Calvin
>>
>

Re: Query about failure conditions in Storm

[Martin Burian <ma...@gmail.com>]

Hi,
the bolt is created and prepared the same way as when the topology is
created. You need to recover your map from the DB yourself, in the
prepare() method of your bolt. Also note that the spout will keep emitting
tuples even when the bolt is dead, so you had better set the
topology.max.pending property to keep the buffers from filling up and the
tuple timeout to accomodate the time your bolt takes to recover.
Hope it helps,
Martin

po 7. 9. 2015 v 8:59 odesílatel Calvin Iyer <ca...@gmail.com> napsal:

> Hi,
>
> I have a query about what happens if a storm bolt dies. My current
> topology is such that I have a Kafka spout that gives out records which are
> counted in a storm bolt and stored in a hashmap in the same bolt and emits
> the aggregated map to another bolt every 5 mins to persist it to a
> database. Now supposing that the bolt dies while having data in the map and
> the task gets reassigned to another worker does the map also get recreated
> or does it use the last snapshot of the map before it failed ?
>
> Thanks,
> Calvin
>