You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Bechennec, Marion" <ma...@dictanova.com> on 2014/09/12 18:51:07 UTC
Multiple groupBy with Trident
Hi,
For one of our applications we are trying to perform multiple groupBy
statements on the same stream, ideally it should look like this :
.stateQuery(...)
.groupBy(new Fields("field"))
.chainedAgg()
.groupBy(new Fields("anotherField"))
.chainedAgg()
.aggregate( ...)
.chainEnd()
.aggregate(....)
.aggregate(....)
.chainEnd();
Obviously this doesn't work, we've tried several things, and came up with
something like this :
.stateQuery(...)
.groupBy(new Fields("field", "anotherField"))
.aggregate( /* Do something */)
.groupBy(new Fields("lemma"))
.chainedAgg()
.aggregate(....)
.aggregate(....)
.chainEnd();
However, this doesn't work either, we are not able to reemit the initial
values on the input stream of the second groupBy statement.
Any thoughts on how can this be accomplished ?
Thank you for your help,
Marion
Re: Multiple groupBy with Trident
Posted by "Bechennec, Marion" <ma...@dictanova.com>.
Thank you for helping :)
We figured out another way for achieving our goal, but we'll try your way
eventually.
2014-09-18 18:00 GMT+02:00 Adam Lewis <ma...@adamlewis.com>:
> You can't emit on two streams in trident, but you can use the DSL to
> "split" a stream into two processing paths, e.g.
>
> Stream *queryResultStream* = someState.stateQuery(...);
>
> Stream firstGroupingAggregate = *queryResultStream*
> .groupBy(/* first grouping */)
> .aggregate(...)
> ;
>
> Stream secondGroupingAggregate = *queryResultStream*
> .groupBy(/* second grouping */)
> .aggregate(...)
> ;
>
> I'm not entirely sure I understand the original request, but if the idea
> of doing different processing on identical streams works for you, then this
> is the idiom for doing that.
>
> On Thu, Sep 18, 2014 at 4:04 AM, Bechennec, Marion <ma...@dictanova.com>
> wrote:
>
>> Hi,
>>
>> Thanks for the answer, but in Trident it doesn't seem possible to emit on
>> a different streams in the same bolt, therefore it wouldn't be possible to
>> combine the output of the 2 streams.
>>
>> 2014-09-15 20:54 GMT+02:00 John Reilly <jr...@inconspicuous.org>:
>>
>>> Can you split the initial stream into 2 identical streams (using a bolt)
>>> and then perform a groupBy on each of the streams and then combine the
>>> output of those 2 groupBys?
>>>
>>> On Fri, Sep 12, 2014 at 9:51 AM, Bechennec, Marion <marion@dictanova.com
>>> > wrote:
>>>
>>>> Hi,
>>>>
>>>> For one of our applications we are trying to perform multiple groupBy
>>>> statements on the same stream, ideally it should look like this :
>>>>
>>>> .stateQuery(...)
>>>> .groupBy(new Fields("field"))
>>>> .chainedAgg()
>>>> .groupBy(new Fields("anotherField"))
>>>> .chainedAgg()
>>>> .aggregate( ...)
>>>> .chainEnd()
>>>> .aggregate(....)
>>>> .aggregate(....)
>>>> .chainEnd();
>>>>
>>>> Obviously this doesn't work, we've tried several things, and came up
>>>> with something like this :
>>>> .stateQuery(...)
>>>> .groupBy(new Fields("field", "anotherField"))
>>>> .aggregate( /* Do something */)
>>>> .groupBy(new Fields("lemma"))
>>>> .chainedAgg()
>>>> .aggregate(....)
>>>> .aggregate(....)
>>>> .chainEnd();
>>>>
>>>> However, this doesn't work either, we are not able to reemit the
>>>> initial values on the input stream of the second groupBy statement.
>>>>
>>>> Any thoughts on how can this be accomplished ?
>>>>
>>>> Thank you for your help,
>>>>
>>>> Marion
>>>>
>>>>
>>>
>>
>
Re: Multiple groupBy with Trident
Posted by Adam Lewis <ma...@adamlewis.com>.
You can't emit on two streams in trident, but you can use the DSL to
"split" a stream into two processing paths, e.g.
Stream *queryResultStream* = someState.stateQuery(...);
Stream firstGroupingAggregate = *queryResultStream*
.groupBy(/* first grouping */)
.aggregate(...)
;
Stream secondGroupingAggregate = *queryResultStream*
.groupBy(/* second grouping */)
.aggregate(...)
;
I'm not entirely sure I understand the original request, but if the idea of
doing different processing on identical streams works for you, then this is
the idiom for doing that.
On Thu, Sep 18, 2014 at 4:04 AM, Bechennec, Marion <ma...@dictanova.com>
wrote:
> Hi,
>
> Thanks for the answer, but in Trident it doesn't seem possible to emit on
> a different streams in the same bolt, therefore it wouldn't be possible to
> combine the output of the 2 streams.
>
> 2014-09-15 20:54 GMT+02:00 John Reilly <jr...@inconspicuous.org>:
>
>> Can you split the initial stream into 2 identical streams (using a bolt)
>> and then perform a groupBy on each of the streams and then combine the
>> output of those 2 groupBys?
>>
>> On Fri, Sep 12, 2014 at 9:51 AM, Bechennec, Marion <ma...@dictanova.com>
>> wrote:
>>
>>> Hi,
>>>
>>> For one of our applications we are trying to perform multiple groupBy
>>> statements on the same stream, ideally it should look like this :
>>>
>>> .stateQuery(...)
>>> .groupBy(new Fields("field"))
>>> .chainedAgg()
>>> .groupBy(new Fields("anotherField"))
>>> .chainedAgg()
>>> .aggregate( ...)
>>> .chainEnd()
>>> .aggregate(....)
>>> .aggregate(....)
>>> .chainEnd();
>>>
>>> Obviously this doesn't work, we've tried several things, and came up
>>> with something like this :
>>> .stateQuery(...)
>>> .groupBy(new Fields("field", "anotherField"))
>>> .aggregate( /* Do something */)
>>> .groupBy(new Fields("lemma"))
>>> .chainedAgg()
>>> .aggregate(....)
>>> .aggregate(....)
>>> .chainEnd();
>>>
>>> However, this doesn't work either, we are not able to reemit the initial
>>> values on the input stream of the second groupBy statement.
>>>
>>> Any thoughts on how can this be accomplished ?
>>>
>>> Thank you for your help,
>>>
>>> Marion
>>>
>>>
>>
>
Re: Multiple groupBy with Trident
Posted by "Bechennec, Marion" <ma...@dictanova.com>.
Hi,
Thanks for the answer, but in Trident it doesn't seem possible to emit on a
different streams in the same bolt, therefore it wouldn't be possible to
combine the output of the 2 streams.
2014-09-15 20:54 GMT+02:00 John Reilly <jr...@inconspicuous.org>:
> Can you split the initial stream into 2 identical streams (using a bolt)
> and then perform a groupBy on each of the streams and then combine the
> output of those 2 groupBys?
>
> On Fri, Sep 12, 2014 at 9:51 AM, Bechennec, Marion <ma...@dictanova.com>
> wrote:
>
>> Hi,
>>
>> For one of our applications we are trying to perform multiple groupBy
>> statements on the same stream, ideally it should look like this :
>>
>> .stateQuery(...)
>> .groupBy(new Fields("field"))
>> .chainedAgg()
>> .groupBy(new Fields("anotherField"))
>> .chainedAgg()
>> .aggregate( ...)
>> .chainEnd()
>> .aggregate(....)
>> .aggregate(....)
>> .chainEnd();
>>
>> Obviously this doesn't work, we've tried several things, and came up with
>> something like this :
>> .stateQuery(...)
>> .groupBy(new Fields("field", "anotherField"))
>> .aggregate( /* Do something */)
>> .groupBy(new Fields("lemma"))
>> .chainedAgg()
>> .aggregate(....)
>> .aggregate(....)
>> .chainEnd();
>>
>> However, this doesn't work either, we are not able to reemit the initial
>> values on the input stream of the second groupBy statement.
>>
>> Any thoughts on how can this be accomplished ?
>>
>> Thank you for your help,
>>
>> Marion
>>
>>
>
Re: Multiple groupBy with Trident
Posted by John Reilly <jr...@inconspicuous.org>.
Can you split the initial stream into 2 identical streams (using a bolt)
and then perform a groupBy on each of the streams and then combine the
output of those 2 groupBys?
On Fri, Sep 12, 2014 at 9:51 AM, Bechennec, Marion <ma...@dictanova.com>
wrote:
> Hi,
>
> For one of our applications we are trying to perform multiple groupBy
> statements on the same stream, ideally it should look like this :
>
> .stateQuery(...)
> .groupBy(new Fields("field"))
> .chainedAgg()
> .groupBy(new Fields("anotherField"))
> .chainedAgg()
> .aggregate( ...)
> .chainEnd()
> .aggregate(....)
> .aggregate(....)
> .chainEnd();
>
> Obviously this doesn't work, we've tried several things, and came up with
> something like this :
> .stateQuery(...)
> .groupBy(new Fields("field", "anotherField"))
> .aggregate( /* Do something */)
> .groupBy(new Fields("lemma"))
> .chainedAgg()
> .aggregate(....)
> .aggregate(....)
> .chainEnd();
>
> However, this doesn't work either, we are not able to reemit the initial
> values on the input stream of the second groupBy statement.
>
> Any thoughts on how can this be accomplished ?
>
> Thank you for your help,
>
> Marion
>
>