You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by anshu shukla <an...@gmail.com> on 2016/04/09 23:14:24 UTC
Message flow in Shuffle grouping
Suppose we have 10 exec/threads of spouts sending tuples to 5 threads of
downstream bolt (say B) .Then my question is -
*Does every thread of spout will uniformly distribute messages to all the 5
threads of downstream nodes .*
*Or *
*Is there any one to one communication btw spouts exec to bolts .*
Very relevant if we want to improve N/W cost by custom scheduler in storm.
--
Thanks & Regards,
Anshu Shukla
Re: Message flow in Shuffle grouping
Posted by anshu shukla <an...@gmail.com>.
Hey Erik
Thanks for reply ,
Actually, I am working on custom scheduler for storm where we can
explicitly specify where to place the thread/executor to particular slot
mapping on a VM . In order to improve N/W communication cost btw threads
.In some papers, I found like if DAG is -
[s0------> b1--------->b2------------>b3] , where every bolt have some
parallelism (say 3 for all ) . connected by shuffle grouping and placement
is done say .
*[VM1(s0------> b1--------->b2------------>b3),VM2(s0------>
b1--------->b2------------>b3),VM3(s0------>
b1--------->b2------------>b3)] (1 thread of each one on diff VMs)*
then the instance of spout *s0 on VM1* will also be sending some tuples to
b1 bolt instance on other VMs .(as per definition of shuffle grouping) .
On Sun, Apr 10, 2016 at 3:03 PM, Erik Weathers <ew...@groupon.com>
wrote:
> hi Anshu. Can you please clarify what is "not clear" to you in the
> description?
>
> On Sun, Apr 10, 2016 at 2:31 AM, anshu shukla <an...@gmail.com>
> wrote:
>
>> Thanks Xiang,
>>
>> I have read it already . But I am not clear about this point -
>>
>>
>> 1. *Local or shuffle grouping: If the target bolt has one or more
>> tasks in the same worker process, tuples will be shuffled to just those
>> in-process tasks. Otherwise, this acts like a normal shuffle grouping*
>>
>>
>> On Sun, Apr 10, 2016 at 6:55 AM, Xiang Wang <xi...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Please have a look at this page:
>>> http://storm.apache.org/releases/0.10.0/Concepts.html
>>>
>>> There are many different grouping methods, and direct grouping may be
>>> what you are looking for.
>>>
>>>
>>> -------------------------------
>>> Xiang Wang PhD Candidate
>>> Database Research Group
>>> School of Computer Science and Engineering
>>> The University of New South Wales
>>> Sydney, Australia
>>>
>>> On Sun, Apr 10, 2016 at 7:14 AM, anshu shukla <an...@gmail.com>
>>> wrote:
>>>
>>>> Suppose we have 10 exec/threads of spouts sending tuples to 5 threads
>>>> of downstream bolt (say B) .Then my question is -
>>>>
>>>> *Does every thread of spout will uniformly distribute messages to all
>>>> the 5 threads of downstream nodes .*
>>>>
>>>> *Or *
>>>>
>>>> *Is there any one to one communication btw spouts exec to bolts .*
>>>>
>>>>
>>>> Very relevant if we want to improve N/W cost by custom scheduler in
>>>> storm.
>>>>
>>>> --
>>>> Thanks & Regards,
>>>> Anshu Shukla
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>
>
--
Thanks & Regards,
Anshu Shukla
Re: Message flow in Shuffle grouping
Posted by Erik Weathers <ew...@groupon.com>.
hi Anshu. Can you please clarify what is "not clear" to you in the
description?
On Sun, Apr 10, 2016 at 2:31 AM, anshu shukla <an...@gmail.com>
wrote:
> Thanks Xiang,
>
> I have read it already . But I am not clear about this point -
>
>
> 1. *Local or shuffle grouping: If the target bolt has one or more
> tasks in the same worker process, tuples will be shuffled to just those
> in-process tasks. Otherwise, this acts like a normal shuffle grouping*
>
>
> On Sun, Apr 10, 2016 at 6:55 AM, Xiang Wang <xi...@gmail.com> wrote:
>
>> Hi,
>>
>> Please have a look at this page:
>> http://storm.apache.org/releases/0.10.0/Concepts.html
>>
>> There are many different grouping methods, and direct grouping may be
>> what you are looking for.
>>
>>
>> -------------------------------
>> Xiang Wang PhD Candidate
>> Database Research Group
>> School of Computer Science and Engineering
>> The University of New South Wales
>> Sydney, Australia
>>
>> On Sun, Apr 10, 2016 at 7:14 AM, anshu shukla <an...@gmail.com>
>> wrote:
>>
>>> Suppose we have 10 exec/threads of spouts sending tuples to 5 threads of
>>> downstream bolt (say B) .Then my question is -
>>>
>>> *Does every thread of spout will uniformly distribute messages to all
>>> the 5 threads of downstream nodes .*
>>>
>>> *Or *
>>>
>>> *Is there any one to one communication btw spouts exec to bolts .*
>>>
>>>
>>> Very relevant if we want to improve N/W cost by custom scheduler in
>>> storm.
>>>
>>> --
>>> Thanks & Regards,
>>> Anshu Shukla
>>>
>>
>>
>
>
> --
> Thanks & Regards,
> Anshu Shukla
>
Re: Message flow in Shuffle grouping
Posted by anshu shukla <an...@gmail.com>.
Thanks Xiang,
I have read it already . But I am not clear about this point -
1. *Local or shuffle grouping: If the target bolt has one or more tasks
in the same worker process, tuples will be shuffled to just those
in-process tasks. Otherwise, this acts like a normal shuffle grouping*
On Sun, Apr 10, 2016 at 6:55 AM, Xiang Wang <xi...@gmail.com> wrote:
> Hi,
>
> Please have a look at this page:
> http://storm.apache.org/releases/0.10.0/Concepts.html
>
> There are many different grouping methods, and direct grouping may be what
> you are looking for.
>
>
> -------------------------------
> Xiang Wang PhD Candidate
> Database Research Group
> School of Computer Science and Engineering
> The University of New South Wales
> Sydney, Australia
>
> On Sun, Apr 10, 2016 at 7:14 AM, anshu shukla <an...@gmail.com>
> wrote:
>
>> Suppose we have 10 exec/threads of spouts sending tuples to 5 threads of
>> downstream bolt (say B) .Then my question is -
>>
>> *Does every thread of spout will uniformly distribute messages to all the
>> 5 threads of downstream nodes .*
>>
>> *Or *
>>
>> *Is there any one to one communication btw spouts exec to bolts .*
>>
>>
>> Very relevant if we want to improve N/W cost by custom scheduler in storm.
>>
>> --
>> Thanks & Regards,
>> Anshu Shukla
>>
>
>
--
Thanks & Regards,
Anshu Shukla
Re: Message flow in Shuffle grouping
Posted by Xiang Wang <xi...@gmail.com>.
Hi,
Please have a look at this page:
http://storm.apache.org/releases/0.10.0/Concepts.html
There are many different grouping methods, and direct grouping may be what
you are looking for.
-------------------------------
Xiang Wang PhD Candidate
Database Research Group
School of Computer Science and Engineering
The University of New South Wales
Sydney, Australia
On Sun, Apr 10, 2016 at 7:14 AM, anshu shukla <an...@gmail.com>
wrote:
> Suppose we have 10 exec/threads of spouts sending tuples to 5 threads of
> downstream bolt (say B) .Then my question is -
>
> *Does every thread of spout will uniformly distribute messages to all the
> 5 threads of downstream nodes .*
>
> *Or *
>
> *Is there any one to one communication btw spouts exec to bolts .*
>
>
> Very relevant if we want to improve N/W cost by custom scheduler in storm.
>
> --
> Thanks & Regards,
> Anshu Shukla
>