You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@storm.apache.org by Sergio Fernández <se...@redlink.co> on 2015/07/14 16:31:20 UTC

usage of aggregate() topology pattern

Hi,

I'm a newbie in many parts of Storm, and I'm 'fighting' against the
aggregation with Trident. I'm using the following topology:

  topology.newStream("tweets", spout)
      .each(new Fields("tweet"), new Analyse(), new Fields("analysis"))
      .aggregate(new Fields("analysis"), new AggregateAnalysis(), new
Fields("aggregation"))
      .each(new Fields("aggregation"), new PrintFilter())

Basically I'm trying to aggregate the sentiment analysis over a batch of
tweets. The spout is a FixedBatchSpout. The functions and the aggregator
look to work fine, the stream is being processed as I expected. But the
problem is it does not stop to process the batch. Reading the documentation
about ReducerAggregator my implementation looks to do whjat I expect. But
still, I can get it aggregating the whole batch.

Anybody has any idea what I'm doing wrong?

All help would be appreciated.

Cheers,

-- 
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 6602747925
e: sergio.fernandez@redlink.co
w: http://redlink.co

Re: usage of aggregate() topology pattern

Posted by Sergio Fernández <se...@redlink.co>.

Does aggregation has a fixed limit?

I've observed that when if goes over 40 reduce operations something happens
in the stream, and the topology starts again to aggregate tuples.

Any hint would be welcomed.

Cheers,


On Tue, Jul 14, 2015 at 4:31 PM, Sergio Fernández <
sergio.fernandez@redlink.co> wrote:

> Hi,
>
> I'm a newbie in many parts of Storm, and I'm 'fighting' against the
> aggregation with Trident. I'm using the following topology:
>
>   topology.newStream("tweets", spout)
>       .each(new Fields("tweet"), new Analyse(), new Fields("analysis"))
>       .aggregate(new Fields("analysis"), new AggregateAnalysis(), new
> Fields("aggregation"))
>       .each(new Fields("aggregation"), new PrintFilter())
>
> Basically I'm trying to aggregate the sentiment analysis over a batch of
> tweets. The spout is a FixedBatchSpout. The functions and the aggregator
> look to work fine, the stream is being processed as I expected. But the
> problem is it does not stop to process the batch. Reading the documentation
> about ReducerAggregator my implementation looks to do whjat I expect. But
> still, I can get it aggregating the whole batch.
>
> Anybody has any idea what I'm doing wrong?
>
> All help would be appreciated.
>
> Cheers,
>
> --
> Sergio Fernández
> Partner Technology Manager
> Redlink GmbH
> m: +43 6602747925
> e: sergio.fernandez@redlink.co
> w: http://redlink.co
>



-- 
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 6602747925
e: sergio.fernandez@redlink.co
w: http://redlink.co