how to consume/reset aggregated bucket

[Rajat Gangwar <ra...@gmail.com>]

Use-Case :

Every hour 100K invoices (entities) are created which are pushed to Storm.
These invoices belong to 'n' users.

So Storm does group aggregation on users, and creates aggregated buckets
per user. Along with sum of invoices, bucket also contains invoices IDs, so
that we know what all invoices got aggregated.
These buckets are getting updated continuously.

Problem : We need to consume these buckets on 2 rules :
                a) bucket has aggregated 1000 invoices, or
                b) bucket created 1 hour ago.

In both cases we need to consume this bucket and do some action on what
invoices are aggregated so far. Before we can consume we need to make sure
that storm should stop updating that bucket and create/update another
bucket for the same customer if new invoices comes.


Is there a way to achieve this in Storm ?


Thanks,
Rajat

Re: how to consume/reset aggregated bucket

[Rajat Gangwar <ra...@gmail.com>]

Let me rephrase my question :

Storm keeps on aggregating same bucket after segregating tuples based on
grouped fields.

- I want to consume all the aggregated content every hour, and reset
aggregation count back to zero. Storm should not be updating this bucket
while consuming.
- Storm then continue aggregating the same bucket.
- After 1 hour repeat the same process.


How can I achieve this in Storm ???
any input would help me a lot.


Thanks,
Rajat


On Tue, Mar 28, 2017 at 12:45 PM, Rajat Gangwar <ra...@gmail.com>
wrote:

> Use-Case :
>
> Every hour 100K invoices (entities) are created which are pushed to Storm.
> These invoices belong to 'n' users.
>
> So Storm does group aggregation on users, and creates aggregated buckets
> per user. Along with sum of invoices, bucket also contains invoices IDs, so
> that we know what all invoices got aggregated.
> These buckets are getting updated continuously.
>
> Problem : We need to consume these buckets on 2 rules :
>                 a) bucket has aggregated 1000 invoices, or
>                 b) bucket created 1 hour ago.
>
> In both cases we need to consume this bucket and do some action on what
> invoices are aggregated so far. Before we can consume we need to make sure
> that storm should stop updating that bucket and create/update another
> bucket for the same customer if new invoices comes.
>
>
> Is there a way to achieve this in Storm ?
>
>
> Thanks,
> Rajat
>
>
>