You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by clay teahouse <cl...@gmail.com> on 2014/10/30 11:57:38 UTC
multi-stream example
Hello All,
Can someone share an example of a bolt with multi stream output, with each
particular output stream going to a particular bolt?
Bolt A =>stream 1 => Bolt B
Bolt A => stream 2 => Bolt C
Bolt A => stream 3 => Bolt D
thanks,
Clay
Re: multi-stream example
Posted by Andrew Xor <an...@gmail.com>.
Well, lots and lots of users initially confuse grouping in storm with
routing tuples... so I thought I'd pour this in regardless ;).
Kindly yours,
Andrew Grammenos
-- PGP PKey --
<https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt
On Thu, Oct 30, 2014 at 2:08 PM, Nathan Leung <nc...@gmail.com> wrote:
> Fair enough I missed the last sentence. But it could be misconstrued since
> he didn't mention groupings at all in his original email.
> On Oct 30, 2014 7:59 AM, "Andrew Xor" <an...@gmail.com> wrote:
>
>> Well, in essence what I said is that if some Bolts are subscribed to a
>> stream they will receive all tuples that flow in that stream regardless of
>> the shuffling you have in each Bolt.
>>
>> Kindly yours,
>>
>> Andrew Grammenos
>>
>> -- PGP PKey --
>> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
>> https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt
>>
>> On Thu, Oct 30, 2014 at 1:54 PM, Nathan Leung <nc...@gmail.com> wrote:
>>
>>> Andrews response is misleading. You can use streams as described by
>>> another user in your other thread. When you create your topology bolt b
>>> subscribed to bolt a stream 1, etc. Then when bolt a emits it can specify
>>> the stream id to emit to, and the tuple will be sent on the appropriate
>>> stream to the appropriate bolt.
>>> On Oct 30, 2014 7:19 AM, "Andrew Xor" <an...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> First of all you have to understand the limitations of Storm
>>>> Processor; by design it emits *all* tuples to *all* subscribed bolts. This
>>>> means that if you have Bolts A,B,C subscribed to a Bolt/Spout D tuples
>>>> emitted by the said storm primitive will go to all subscribed Bolts. In
>>>> effect the grouping type is *not* a routing scheme but a parallelism scheme
>>>> for distributing the tuples received in the bolt to its workers. To achieve
>>>> the said functionality you have to get creative; for example you could
>>>> subscribe these bolts using three distinct direct streams, one for each and
>>>> then conditionally emit to each stream the tuples you want to be processed
>>>> by each Bolt.
>>>>
>>>> Hope this helped.
>>>>
>>>> Kindly yours,
>>>>
>>>> Andrew Grammenos
>>>>
>>>> -- PGP PKey --
>>>> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
>>>> https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt
>>>>
>>>> On Thu, Oct 30, 2014 at 12:57 PM, clay teahouse <clayteahouse@gmail.com
>>>> > wrote:
>>>>
>>>>> Hello All,
>>>>> Can someone share an example of a bolt with multi stream output, with
>>>>> each particular output stream going to a particular bolt?
>>>>>
>>>>> Bolt A =>stream 1 => Bolt B
>>>>> Bolt A => stream 2 => Bolt C
>>>>> Bolt A => stream 3 => Bolt D
>>>>>
>>>>> thanks,
>>>>> Clay
>>>>>
>>>>
>>>>
>>
Re: multi-stream example
Posted by Nathan Leung <nc...@gmail.com>.
Fair enough I missed the last sentence. But it could be misconstrued since
he didn't mention groupings at all in his original email.
On Oct 30, 2014 7:59 AM, "Andrew Xor" <an...@gmail.com> wrote:
> Well, in essence what I said is that if some Bolts are subscribed to a
> stream they will receive all tuples that flow in that stream regardless of
> the shuffling you have in each Bolt.
>
> Kindly yours,
>
> Andrew Grammenos
>
> -- PGP PKey --
> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
> https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt
>
> On Thu, Oct 30, 2014 at 1:54 PM, Nathan Leung <nc...@gmail.com> wrote:
>
>> Andrews response is misleading. You can use streams as described by
>> another user in your other thread. When you create your topology bolt b
>> subscribed to bolt a stream 1, etc. Then when bolt a emits it can specify
>> the stream id to emit to, and the tuple will be sent on the appropriate
>> stream to the appropriate bolt.
>> On Oct 30, 2014 7:19 AM, "Andrew Xor" <an...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> First of all you have to understand the limitations of Storm Processor;
>>> by design it emits *all* tuples to *all* subscribed bolts. This means that
>>> if you have Bolts A,B,C subscribed to a Bolt/Spout D tuples emitted by the
>>> said storm primitive will go to all subscribed Bolts. In effect the
>>> grouping type is *not* a routing scheme but a parallelism scheme for
>>> distributing the tuples received in the bolt to its workers. To achieve the
>>> said functionality you have to get creative; for example you could
>>> subscribe these bolts using three distinct direct streams, one for each and
>>> then conditionally emit to each stream the tuples you want to be processed
>>> by each Bolt.
>>>
>>> Hope this helped.
>>>
>>> Kindly yours,
>>>
>>> Andrew Grammenos
>>>
>>> -- PGP PKey --
>>> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
>>> https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt
>>>
>>> On Thu, Oct 30, 2014 at 12:57 PM, clay teahouse <cl...@gmail.com>
>>> wrote:
>>>
>>>> Hello All,
>>>> Can someone share an example of a bolt with multi stream output, with
>>>> each particular output stream going to a particular bolt?
>>>>
>>>> Bolt A =>stream 1 => Bolt B
>>>> Bolt A => stream 2 => Bolt C
>>>> Bolt A => stream 3 => Bolt D
>>>>
>>>> thanks,
>>>> Clay
>>>>
>>>
>>>
>
Re: multi-stream example
Posted by Andrew Xor <an...@gmail.com>.
Well, in essence what I said is that if some Bolts are subscribed to a
stream they will receive all tuples that flow in that stream regardless of
the shuffling you have in each Bolt.
Kindly yours,
Andrew Grammenos
-- PGP PKey --
<https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt
On Thu, Oct 30, 2014 at 1:54 PM, Nathan Leung <nc...@gmail.com> wrote:
> Andrews response is misleading. You can use streams as described by
> another user in your other thread. When you create your topology bolt b
> subscribed to bolt a stream 1, etc. Then when bolt a emits it can specify
> the stream id to emit to, and the tuple will be sent on the appropriate
> stream to the appropriate bolt.
> On Oct 30, 2014 7:19 AM, "Andrew Xor" <an...@gmail.com> wrote:
>
>> Hi,
>>
>> First of all you have to understand the limitations of Storm Processor;
>> by design it emits *all* tuples to *all* subscribed bolts. This means that
>> if you have Bolts A,B,C subscribed to a Bolt/Spout D tuples emitted by the
>> said storm primitive will go to all subscribed Bolts. In effect the
>> grouping type is *not* a routing scheme but a parallelism scheme for
>> distributing the tuples received in the bolt to its workers. To achieve the
>> said functionality you have to get creative; for example you could
>> subscribe these bolts using three distinct direct streams, one for each and
>> then conditionally emit to each stream the tuples you want to be processed
>> by each Bolt.
>>
>> Hope this helped.
>>
>> Kindly yours,
>>
>> Andrew Grammenos
>>
>> -- PGP PKey --
>> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
>> https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt
>>
>> On Thu, Oct 30, 2014 at 12:57 PM, clay teahouse <cl...@gmail.com>
>> wrote:
>>
>>> Hello All,
>>> Can someone share an example of a bolt with multi stream output, with
>>> each particular output stream going to a particular bolt?
>>>
>>> Bolt A =>stream 1 => Bolt B
>>> Bolt A => stream 2 => Bolt C
>>> Bolt A => stream 3 => Bolt D
>>>
>>> thanks,
>>> Clay
>>>
>>
>>
Re: multi-stream example
Posted by Nathan Leung <nc...@gmail.com>.
Andrews response is misleading. You can use streams as described by another
user in your other thread. When you create your topology bolt b subscribed
to bolt a stream 1, etc. Then when bolt a emits it can specify the stream
id to emit to, and the tuple will be sent on the appropriate stream to the
appropriate bolt.
On Oct 30, 2014 7:19 AM, "Andrew Xor" <an...@gmail.com> wrote:
> Hi,
>
> First of all you have to understand the limitations of Storm Processor;
> by design it emits *all* tuples to *all* subscribed bolts. This means that
> if you have Bolts A,B,C subscribed to a Bolt/Spout D tuples emitted by the
> said storm primitive will go to all subscribed Bolts. In effect the
> grouping type is *not* a routing scheme but a parallelism scheme for
> distributing the tuples received in the bolt to its workers. To achieve the
> said functionality you have to get creative; for example you could
> subscribe these bolts using three distinct direct streams, one for each and
> then conditionally emit to each stream the tuples you want to be processed
> by each Bolt.
>
> Hope this helped.
>
> Kindly yours,
>
> Andrew Grammenos
>
> -- PGP PKey --
> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
> https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt
>
> On Thu, Oct 30, 2014 at 12:57 PM, clay teahouse <cl...@gmail.com>
> wrote:
>
>> Hello All,
>> Can someone share an example of a bolt with multi stream output, with
>> each particular output stream going to a particular bolt?
>>
>> Bolt A =>stream 1 => Bolt B
>> Bolt A => stream 2 => Bolt C
>> Bolt A => stream 3 => Bolt D
>>
>> thanks,
>> Clay
>>
>
>
Re: multi-stream example
Posted by Andrew Xor <an...@gmail.com>.
Hi,
First of all you have to understand the limitations of Storm Processor; by
design it emits *all* tuples to *all* subscribed bolts. This means that if
you have Bolts A,B,C subscribed to a Bolt/Spout D tuples emitted by the
said storm primitive will go to all subscribed Bolts. In effect the
grouping type is *not* a routing scheme but a parallelism scheme for
distributing the tuples received in the bolt to its workers. To achieve the
said functionality you have to get creative; for example you could
subscribe these bolts using three distinct direct streams, one for each and
then conditionally emit to each stream the tuples you want to be processed
by each Bolt.
Hope this helped.
Kindly yours,
Andrew Grammenos
-- PGP PKey --
<https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>
https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt
On Thu, Oct 30, 2014 at 12:57 PM, clay teahouse <cl...@gmail.com>
wrote:
> Hello All,
> Can someone share an example of a bolt with multi stream output, with each
> particular output stream going to a particular bolt?
>
> Bolt A =>stream 1 => Bolt B
> Bolt A => stream 2 => Bolt C
> Bolt A => stream 3 => Bolt D
>
> thanks,
> Clay
>