You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Spico Florin <sp...@gmail.com> on 2016/02/15 12:50:59 UTC
localOrShuffleGrouping load balanced tuple distribution
Hello!
Suppose that I have the following scenario
1. One spout
2. One bolt with hintParallelism set to 4
3. Bolt connected with the spout with localOrShuffleGrouping
4. 2 slots available
5. We use the the default schedulder (round-robin)
Give the above scenario, the distribution over the workers will be:
worker 1: spout, bolt, bolt
worker 2: bolt, bolt
Here are my questions:
1. Will the spout uses the 4 bolts?
2. Will the data distributed evenly between these bolts?
3. Due to localorShuffleGrouping method, only the local bolts will be used
(the ones from the worker 1) or all the workers?
I look forward for your answers.
Thank you.
Florin
Re: localOrShuffleGrouping load balanced tuple distribution
Posted by Bobby Evans <ev...@yahoo-inc.com>.
We are working on way to try to improve this by having the grouping be aware of the load downstream and try to route tuples locally unless it is overloaded and then send them remotely.
- Bobby
On Tuesday, February 16, 2016 8:27 AM, Nathan Leung <nc...@gmail.com> wrote:
My guess is this only happens because you have one executor per worker so nothing is local to the spout anyways. If your component has number of executors less than your number of workers it's often better to use shuffle grouping. In your two worker example, half the tuples would stay local anyways though obviously this decreases as you add workers.On Feb 16, 2016 5:23 AM, "Spico Florin" <sp...@gmail.com> wrote:
Hi, Nathan! Thank you for your answer. You are right: the only bolts that are used will be the ones located nearby the spout. The other two will be unused. If we increase the number of workers to 5, then it will behave as shuffleGrouping (distributing evenly among the bolts). I got it! Regards, Florin
On Mon, Feb 15, 2016 at 3:56 PM, Nathan Leung <nc...@gmail.com> wrote:
It will use the local bolts.
On Mon, Feb 15, 2016 at 8:32 AM, Spico Florin <sp...@gmail.com> wrote:
Hello! Suppose that I have the following scenario1. One spout2. One bolt with hintParallelism set to 43. Bolt connected with the spout with localOrShuffleGrouping4. 2 slots available5. We use the the default schedulder (round-robin)
Give the above scenario, the distribution over the workers will be:worker 1: spout, bolt, boltworker 2: bolt, bolt
Here are my questions:1. Will the spout uses the 4 bolts?2. Will the data distributed evenly between these bolts?3. Due to localorShuffleGrouping method, only the local bolts will be used (the ones from the worker 1) or all the workers?
I look forward for your answers.Thank you. Florin
Re: localOrShuffleGrouping load balanced tuple distribution
Posted by Nathan Leung <nc...@gmail.com>.
My guess is this only happens because you have one executor per worker so
nothing is local to the spout anyways. If your component has number of
executors less than your number of workers it's often better to use shuffle
grouping. In your two worker example, half the tuples would stay local
anyways though obviously this decreases as you add workers.
On Feb 16, 2016 5:23 AM, "Spico Florin" <sp...@gmail.com> wrote:
> Hi, Nathan!
> Thank you for your answer. You are right: the only bolts that are used
> will be the ones located nearby the spout. The other two will be unused. If
> we increase the number of workers to 5, then it will behave as
> shuffleGrouping (distributing evenly among the bolts).
> I got it!
> Regards,
> Florin
>
> On Mon, Feb 15, 2016 at 3:56 PM, Nathan Leung <nc...@gmail.com> wrote:
>
>> It will use the local bolts.
>>
>> On Mon, Feb 15, 2016 at 8:32 AM, Spico Florin <sp...@gmail.com>
>> wrote:
>>
>>>
>>> Hello!
>>> Suppose that I have the following scenario
>>> 1. One spout
>>> 2. One bolt with hintParallelism set to 4
>>> 3. Bolt connected with the spout with localOrShuffleGrouping
>>> 4. 2 slots available
>>> 5. We use the the default schedulder (round-robin)
>>>
>>> Give the above scenario, the distribution over the workers will be:
>>> worker 1: spout, bolt, bolt
>>> worker 2: bolt, bolt
>>>
>>> Here are my questions:
>>> 1. Will the spout uses the 4 bolts?
>>> 2. Will the data distributed evenly between these bolts?
>>> 3. Due to localorShuffleGrouping method, only the local bolts will be
>>> used (the ones from the worker 1) or all the workers?
>>>
>>> I look forward for your answers.
>>> Thank you.
>>> Florin
>>>
>>>
>>>
>>>
>>>
>>
>
Re: localOrShuffleGrouping load balanced tuple distribution
Posted by Spico Florin <sp...@gmail.com>.
Hi, Nathan!
Thank you for your answer. You are right: the only bolts that are used
will be the ones located nearby the spout. The other two will be unused. If
we increase the number of workers to 5, then it will behave as
shuffleGrouping (distributing evenly among the bolts).
I got it!
Regards,
Florin
On Mon, Feb 15, 2016 at 3:56 PM, Nathan Leung <nc...@gmail.com> wrote:
> It will use the local bolts.
>
> On Mon, Feb 15, 2016 at 8:32 AM, Spico Florin <sp...@gmail.com>
> wrote:
>
>>
>> Hello!
>> Suppose that I have the following scenario
>> 1. One spout
>> 2. One bolt with hintParallelism set to 4
>> 3. Bolt connected with the spout with localOrShuffleGrouping
>> 4. 2 slots available
>> 5. We use the the default schedulder (round-robin)
>>
>> Give the above scenario, the distribution over the workers will be:
>> worker 1: spout, bolt, bolt
>> worker 2: bolt, bolt
>>
>> Here are my questions:
>> 1. Will the spout uses the 4 bolts?
>> 2. Will the data distributed evenly between these bolts?
>> 3. Due to localorShuffleGrouping method, only the local bolts will be
>> used (the ones from the worker 1) or all the workers?
>>
>> I look forward for your answers.
>> Thank you.
>> Florin
>>
>>
>>
>>
>>
>
Re: localOrShuffleGrouping load balanced tuple distribution
Posted by Nathan Leung <nc...@gmail.com>.
It will use the local bolts.
On Mon, Feb 15, 2016 at 8:32 AM, Spico Florin <sp...@gmail.com> wrote:
>
> Hello!
> Suppose that I have the following scenario
> 1. One spout
> 2. One bolt with hintParallelism set to 4
> 3. Bolt connected with the spout with localOrShuffleGrouping
> 4. 2 slots available
> 5. We use the the default schedulder (round-robin)
>
> Give the above scenario, the distribution over the workers will be:
> worker 1: spout, bolt, bolt
> worker 2: bolt, bolt
>
> Here are my questions:
> 1. Will the spout uses the 4 bolts?
> 2. Will the data distributed evenly between these bolts?
> 3. Due to localorShuffleGrouping method, only the local bolts will be used
> (the ones from the worker 1) or all the workers?
>
> I look forward for your answers.
> Thank you.
> Florin
>
>
>
>
>
Fwd: localOrShuffleGrouping load balanced tuple distribution
Posted by Spico Florin <sp...@gmail.com>.
Hello!
Suppose that I have the following scenario
1. One spout
2. One bolt with hintParallelism set to 4
3. Bolt connected with the spout with localOrShuffleGrouping
4. 2 slots available
5. We use the the default schedulder (round-robin)
Give the above scenario, the distribution over the workers will be:
worker 1: spout, bolt, bolt
worker 2: bolt, bolt
Here are my questions:
1. Will the spout uses the 4 bolts?
2. Will the data distributed evenly between these bolts?
3. Due to localorShuffleGrouping method, only the local bolts will be used
(the ones from the worker 1) or all the workers?
I look forward for your answers.
Thank you.
Florin