Stateful bolts

[Abhishek Raj <ab...@saavn.com>]

Hi. I was looking into state management
<http://storm.apache.org/releases/1.0.2/State-checkpointing.html> which
looks like a really cool feature but I am wondering if it'd be right for my
use case. One of the bolts does bulk updates to database and stores the
update information temporarily in hashmap. Once the update is done, the map
is cleared. If the worker crashes, the update data is lost so I want to
persist update information across restarts using RedisKeyValueState
<https://github.com/apache/storm/blob/99a11cb2890b273520cd29cffb5806f5111abeb6/external/storm-redis/src/main/java/org/apache/storm/redis/state/RedisKeyValueState.java>.
I could use get/put to modify individual values in KeyValueState but the
interface doesn't expose any method to clear *all *values in the
KeyValueState. So I am wondering if state management is not the right tool
here and what's the appropriate use of state management. Any insight on
this would be great.

Thanks.

-- 
Abhishek

Re: Stateful bolts

[Arun Mahadevan <ar...@apache.org>]

Hi Abhishek,

 

Right now delete/clear is not supported, so you need to workaround this by putting ‘null’ values. 

 

Clear/delete will be added in future. There’s a pending patch to support delete (https://github.com/apache/storm/pull/1470).

 

Thanks,

Arun

 

From: Abhishek Raj <ab...@saavn.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Thursday, November 3, 2016 at 8:12 PM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Stateful bolts

 

Hi. I was looking into state management which looks like a really cool feature but I am wondering if it'd be right for my use case. One of the bolts does bulk updates to database and stores the update information temporarily in hashmap. Once the update is done, the map is cleared. If the worker crashes, the update data is lost so I want to persist update information across restarts using RedisKeyValueState. I could use get/put to modify individual values in KeyValueState but the interface doesn't expose any method to clear all values in the KeyValueState. So I am wondering if state management is not the right tool here and what's the appropriate use of state management. Any insight on this would be great. 

 

Thanks. 
 

-- 

Abhishek