Storm processing in bursts

[Vladi Feigin <vl...@gmail.com>]

Hi All,

We're planing to develop a couple of topologies, which process the data in
bursts.
The data producer accumulates the data in files for a few minutes and after
it send the bulks to Kafka then the storm topology processes it.
Do you see any problem with this approach?
Thank you in advance,
Vladi

Re: Storm processing in bursts

[Eugene <ed...@gmail.com>]

Hi Vladi,
Looks like you want to do batch processing or micro-batch processing over
streaming system. I don't see any problem with that, a lot of people are
using Storm for batch processing. Storm is doing micro-batches itself in
Trident, where spout wait to  collect data for batch (albeit not minutes)
and send the whole batch down to topology.



On Sun, Sep 7, 2014 at 5:52 PM, Tero Paananen <te...@gmail.com>
wrote:

> > The data producer accumulates the data in files for a few minutes and
> after
> > it send the bulks to Kafka then the storm topology processes it.
> > Do you see any problem with this approach?
>
> Assuming the data is processed fast (i.e. in seconds, rather than
> minutes), you're wasting your resources.
>
> Your servers are doing nothing for almost all the time, and then doing
> a lot of work for a short period of time. That's going to be a pretty
> inefficient way of using server resources.
>
> -TPP
>



-- 
Eugene Dvorkin
Software Engineer
New York City Storm User Group - organizer
WebMD
email: edvorkin@gmail.com
phone: 646-250-9649
eugenedvorkin.com
Connect with me on:
LinkedIn <http://www.linkedin.com/in/eugenedvorkin> Twitter
<http://twitter.com/edvorkin>

Re: Storm processing in bursts

[Tero Paananen <te...@gmail.com>]

> The data producer accumulates the data in files for a few minutes and after
> it send the bulks to Kafka then the storm topology processes it.
> Do you see any problem with this approach?

Assuming the data is processed fast (i.e. in seconds, rather than
minutes), you're wasting your resources.

Your servers are doing nothing for almost all the time, and then doing
a lot of work for a short period of time. That's going to be a pretty
inefficient way of using server resources.

-TPP