You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Quentin de G." <qu...@gmail.com> on 2014/03/27 14:45:02 UTC

Workers / Tasks management in case of failure

Hello everyone,

I'm building a topology in which I'm counting connected devices. Each
device can send a start and a stop message, both with a same unique ID (to
match the device).

Due to the high throughput of messages, I'm storing a state (in my case, a
list of connected devices) in RAM, shared across all workers, to check and
count devices.
I'm also storing each start message in HBase, using Put in bulk (and also
removing when I got end messages). I don't want to count using HBase (too
slow).

I'd like to be able, in case of a worker / task failure, to restore the
state in RAM by reading the values that are into HBase.

Problem is, I have no idea on how to implement this with Trident.

I found this PDF:
http://didata.us/assets.d/DiData-StormRedis-Portfolio.pdf, that seems
to be a good way of doing it, but only with Storm. I need to
achieve exactly-once processing thanks to Trident.

My questions is, can I achieve all this using Trident ? Is the idea of
combining RAM + HBase is wrong ?

Many thanks,
Quentin