New stream out of already processed and stored data

[Daniela Stoiber <da...@gmail.com>]

Hello

I would like to process, join and aggregate data (with Storm Bolts) and
store this data with another Bolt into a database (e.g. Redis) to create a
new stream out of this data stored in Redis to do further aggregation on
it.

Would it be possible to create a new stream out of the processed data
stored in Redis with a new Storm Spout or would I have to use another tool
to create the new stream or would it in general be easier to do the
aggregations in Redis itself without the use of Storm.

Thank you very much in advance.

Best regards,
Daniela

Re: New stream out of already processed and stored data

[Bobby Evans <ev...@yahoo-inc.com.INVALID>]

It all depends on how you want to do it, and a lot of it depends on who will be consuming the data downstream.  You can have a bolt that all it does it publish to redis, or kafka, or mysql, or what ever other store you want.  You can then choose to either have a separate spout that consumes this data from the store, or you can just us the same stream that was going into the storage bolt, and use that to do your other aggregations.  Having a separate spout is useful if multiple different groups want to consume that data.  I.e. Group a takes the data and cleans it up, but group b handles finances and counts the money made, and group c handles targeting and helps see what people like.  Having all three groups coordinate writing and deploying a single topology is a pain, so having an intermediate message broker can provide a good separation between the different groups. 
 - Bobby 

    On Monday, January 4, 2016 1:08 AM, Daniela Stoiber <da...@gmail.com> wrote:
 

 Hello

I would like to process, join and aggregate data (with Storm Bolts) and
store this data with another Bolt into a database (e.g. Redis) to create a
new stream out of this data stored in Redis to do further aggregation on
it.

Would it be possible to create a new stream out of the processed data
stored in Redis with a new Storm Spout or would I have to use another tool
to create the new stream or would it in general be easier to do the
aggregations in Redis itself without the use of Storm.

Thank you very much in advance.

Best regards,
Daniela