You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by bijoy deb <bi...@gmail.com> on 2014/03/21 10:50:27 UTC
Multiple Storm components getting assigned to same worker slot
despite of free slots being available
Hi,
I am running a topology using Storm (version 0.9.1),on a cluster of 3 nodes
(3x4=12 slots). My topology has 1 spout(parallelism=2),bolt A
(parallelism=2),bolt B (parallelism=1) and bolt C(parallelism=1).Number of
tasks (numTasks) for each component is default(1).Number of workers is set
as 5.
Given above scenario,when I submit the topology,I can see 5 slots are used
up and 7 are free (out of 12).But still one instance of the spout and bolt
C are going to the same worker slot (e.g port 6703 of node 1). Shouldn't
Storm be ensuring that components are assigned to distinct unused slots as
long as there are empty slots available? Or is there some configuration
that I have missed or misconfigured here?
I have pasted the screenshot of Storm UI below for reference.
Thanks
Bijoy
Re: Multiple Storm components getting assigned to same worker slot
despite of free slots being available
Posted by Srinath C <sr...@gmail.com>.
There configs are interesting but undocumented on wiki.
Thanks for the info.
On Fri, Mar 21, 2014 at 10:37 PM, Drew Goya <dr...@gradientx.com> wrote:
> Take a look at "topology.optimize" and "storm.scheduler"
>
> I had the same issue and I found that setting topology.optimize to false
> and storm.scheduler to "backtype.storm.scheduler.EvenScheduler" gave me the
> even distribution of components I was looking for.
>
>
> On Fri, Mar 21, 2014 at 2:50 AM, bijoy deb <bi...@gmail.com>wrote:
>
>> Hi,
>>
>> I am running a topology using Storm (version 0.9.1),on a cluster of 3
>> nodes (3x4=12 slots). My topology has 1 spout(parallelism=2),bolt A
>> (parallelism=2),bolt B (parallelism=1) and bolt C(parallelism=1).Number of
>> tasks (numTasks) for each component is default(1).Number of workers is set
>> as 5.
>>
>> Given above scenario,when I submit the topology,I can see 5 slots are
>> used up and 7 are free (out of 12).But still one instance of the spout and
>> bolt C are going to the same worker slot (e.g port 6703 of node 1).
>> Shouldn't Storm be ensuring that components are assigned to distinct unused
>> slots as long as there are empty slots available? Or is there some
>> configuration that I have missed or misconfigured here?
>>
>> I have pasted the screenshot of Storm UI below for reference.
>>
>> Thanks
>> Bijoy
>>
>>
>
RE: Multiple Storm components getting assigned to same worker slot
despite of free slots being available
Posted by Simon Cooper <si...@featurespace.co.uk>.
EvenScheduler doesn't seem to do the trick. I've set the scheduler to be EvenScheduler, but the two topologies we've got are still being assigned to the same supervisor, when there's 3 possible supervisors to assign to. It's hard to tell what exactly EvenScheduler is doing, is there some specification of what exactly EvenScheduler and DefaultScheduler do somewhere?
From: bijoy deb [mailto:bijoy.computers@gmail.com]
Sent: 22 March 2014 05:39
To: user@storm.incubator.apache.org
Subject: Re: Multiple Storm components getting assigned to same worker slot despite of free slots being available
Thanks Nathan.So, I believe setting storm.scheduler to EvenScheduler,as suggested by Drew should do the trick?
However, I still have one doubt.With reference to my use case, what I was looking for is that each component (spout/bolt instance) should get assigned to different slot,since I have free slots available;but instead multiple instances are getting assigned to same slot. Does setting the scheduler to EvenScheduler ensure even distribution of tasks across slots within a single worker machine also,or does it just ensure even distribution across multiple worker machines only.
Also,shouldn't this scheduler property setting be there by default,since we would always like to assign tasks to empty slots first?
Thanks
Bijoy
On Sat, Mar 22, 2014 at 10:57 AM, Nathan Marz <na...@nathanmarz.com>> wrote:
topology.optimize doesn't do anything at the moment. It was something planned for in the early days but turned out to be unecessary.
On Fri, Mar 21, 2014 at 9:00 PM, bijoy deb <bi...@gmail.com>> wrote:
Thanks Drew.I am going to try those options and see if that helps.
Thanks
Bijoy
On Fri, Mar 21, 2014 at 10:37 PM, Drew Goya <dr...@gradientx.com>> wrote:
Take a look at "topology.optimize" and "storm.scheduler"
I had the same issue and I found that setting topology.optimize to false and storm.scheduler to "backtype.storm.scheduler.EvenScheduler" gave me the even distribution of components I was looking for.
On Fri, Mar 21, 2014 at 2:50 AM, bijoy deb <bi...@gmail.com>> wrote:
Hi,
I am running a topology using Storm (version 0.9.1),on a cluster of 3 nodes (3x4=12 slots). My topology has 1 spout(parallelism=2),bolt A (parallelism=2),bolt B (parallelism=1) and bolt C(parallelism=1).Number of tasks (numTasks) for each component is default(1).Number of workers is set as 5.
Given above scenario,when I submit the topology,I can see 5 slots are used up and 7 are free (out of 12).But still one instance of the spout and bolt C are going to the same worker slot (e.g port 6703 of node 1). Shouldn't Storm be ensuring that components are assigned to distinct unused slots as long as there are empty slots available? Or is there some configuration that I have missed or misconfigured here?
I have pasted the screenshot of Storm UI below for reference.
Thanks
Bijoy
--
Twitter: @nathanmarz
http://nathanmarz.com<http://nathanmarz.com/>
Re: Multiple Storm components getting assigned to same worker slot
despite of free slots being available
Posted by bijoy deb <bi...@gmail.com>.
Thanks Nathan.So, I believe setting storm.scheduler to EvenScheduler,as
suggested by Drew should do the trick?
However, I still have one doubt.With reference to my use case, what I was
looking for is that each component (spout/bolt instance) should get
assigned to different slot,since I have free slots available;but instead
multiple instances are getting assigned to same slot. Does setting the
scheduler to EvenScheduler ensure even distribution of tasks across slots
within a single worker machine also,or does it just ensure even
distribution across multiple worker machines only.
Also,shouldn't this scheduler property setting be there by default,since we
would always like to assign tasks to empty slots first?
Thanks
Bijoy
On Sat, Mar 22, 2014 at 10:57 AM, Nathan Marz <na...@nathanmarz.com> wrote:
> topology.optimize doesn't do anything at the moment. It was something
> planned for in the early days but turned out to be unecessary.
>
>
> On Fri, Mar 21, 2014 at 9:00 PM, bijoy deb <bi...@gmail.com>wrote:
>
>> Thanks Drew.I am going to try those options and see if that helps.
>>
>> Thanks
>> Bijoy
>>
>>
>> On Fri, Mar 21, 2014 at 10:37 PM, Drew Goya <dr...@gradientx.com> wrote:
>>
>>> Take a look at "topology.optimize" and "storm.scheduler"
>>>
>>> I had the same issue and I found that setting topology.optimize to false
>>> and storm.scheduler to "backtype.storm.scheduler.EvenScheduler" gave me the
>>> even distribution of components I was looking for.
>>>
>>>
>>> On Fri, Mar 21, 2014 at 2:50 AM, bijoy deb <bi...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I am running a topology using Storm (version 0.9.1),on a cluster of 3
>>>> nodes (3x4=12 slots). My topology has 1 spout(parallelism=2),bolt A
>>>> (parallelism=2),bolt B (parallelism=1) and bolt C(parallelism=1).Number of
>>>> tasks (numTasks) for each component is default(1).Number of workers is set
>>>> as 5.
>>>>
>>>> Given above scenario,when I submit the topology,I can see 5 slots are
>>>> used up and 7 are free (out of 12).But still one instance of the spout and
>>>> bolt C are going to the same worker slot (e.g port 6703 of node 1).
>>>> Shouldn't Storm be ensuring that components are assigned to distinct unused
>>>> slots as long as there are empty slots available? Or is there some
>>>> configuration that I have missed or misconfigured here?
>>>>
>>>> I have pasted the screenshot of Storm UI below for reference.
>>>>
>>>> Thanks
>>>> Bijoy
>>>>
>>>>
>>>
>>
>
>
> --
> Twitter: @nathanmarz
> http://nathanmarz.com
>
Re: Multiple Storm components getting assigned to same worker slot
despite of free slots being available
Posted by Nathan Marz <na...@nathanmarz.com>.
topology.optimize doesn't do anything at the moment. It was something
planned for in the early days but turned out to be unecessary.
On Fri, Mar 21, 2014 at 9:00 PM, bijoy deb <bi...@gmail.com>wrote:
> Thanks Drew.I am going to try those options and see if that helps.
>
> Thanks
> Bijoy
>
>
> On Fri, Mar 21, 2014 at 10:37 PM, Drew Goya <dr...@gradientx.com> wrote:
>
>> Take a look at "topology.optimize" and "storm.scheduler"
>>
>> I had the same issue and I found that setting topology.optimize to false
>> and storm.scheduler to "backtype.storm.scheduler.EvenScheduler" gave me the
>> even distribution of components I was looking for.
>>
>>
>> On Fri, Mar 21, 2014 at 2:50 AM, bijoy deb <bi...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I am running a topology using Storm (version 0.9.1),on a cluster of 3
>>> nodes (3x4=12 slots). My topology has 1 spout(parallelism=2),bolt A
>>> (parallelism=2),bolt B (parallelism=1) and bolt C(parallelism=1).Number of
>>> tasks (numTasks) for each component is default(1).Number of workers is set
>>> as 5.
>>>
>>> Given above scenario,when I submit the topology,I can see 5 slots are
>>> used up and 7 are free (out of 12).But still one instance of the spout and
>>> bolt C are going to the same worker slot (e.g port 6703 of node 1).
>>> Shouldn't Storm be ensuring that components are assigned to distinct unused
>>> slots as long as there are empty slots available? Or is there some
>>> configuration that I have missed or misconfigured here?
>>>
>>> I have pasted the screenshot of Storm UI below for reference.
>>>
>>> Thanks
>>> Bijoy
>>>
>>>
>>
>
--
Twitter: @nathanmarz
http://nathanmarz.com
Re: Multiple Storm components getting assigned to same worker slot
despite of free slots being available
Posted by bijoy deb <bi...@gmail.com>.
Thanks Drew.I am going to try those options and see if that helps.
Thanks
Bijoy
On Fri, Mar 21, 2014 at 10:37 PM, Drew Goya <dr...@gradientx.com> wrote:
> Take a look at "topology.optimize" and "storm.scheduler"
>
> I had the same issue and I found that setting topology.optimize to false
> and storm.scheduler to "backtype.storm.scheduler.EvenScheduler" gave me the
> even distribution of components I was looking for.
>
>
> On Fri, Mar 21, 2014 at 2:50 AM, bijoy deb <bi...@gmail.com>wrote:
>
>> Hi,
>>
>> I am running a topology using Storm (version 0.9.1),on a cluster of 3
>> nodes (3x4=12 slots). My topology has 1 spout(parallelism=2),bolt A
>> (parallelism=2),bolt B (parallelism=1) and bolt C(parallelism=1).Number of
>> tasks (numTasks) for each component is default(1).Number of workers is set
>> as 5.
>>
>> Given above scenario,when I submit the topology,I can see 5 slots are
>> used up and 7 are free (out of 12).But still one instance of the spout and
>> bolt C are going to the same worker slot (e.g port 6703 of node 1).
>> Shouldn't Storm be ensuring that components are assigned to distinct unused
>> slots as long as there are empty slots available? Or is there some
>> configuration that I have missed or misconfigured here?
>>
>> I have pasted the screenshot of Storm UI below for reference.
>>
>> Thanks
>> Bijoy
>>
>>
>
Re: Multiple Storm components getting assigned to same worker slot
despite of free slots being available
Posted by Drew Goya <dr...@gradientx.com>.
Take a look at "topology.optimize" and "storm.scheduler"
I had the same issue and I found that setting topology.optimize to false
and storm.scheduler to "backtype.storm.scheduler.EvenScheduler" gave me the
even distribution of components I was looking for.
On Fri, Mar 21, 2014 at 2:50 AM, bijoy deb <bi...@gmail.com>wrote:
> Hi,
>
> I am running a topology using Storm (version 0.9.1),on a cluster of 3
> nodes (3x4=12 slots). My topology has 1 spout(parallelism=2),bolt A
> (parallelism=2),bolt B (parallelism=1) and bolt C(parallelism=1).Number of
> tasks (numTasks) for each component is default(1).Number of workers is set
> as 5.
>
> Given above scenario,when I submit the topology,I can see 5 slots are used
> up and 7 are free (out of 12).But still one instance of the spout and bolt
> C are going to the same worker slot (e.g port 6703 of node 1). Shouldn't
> Storm be ensuring that components are assigned to distinct unused slots as
> long as there are empty slots available? Or is there some configuration
> that I have missed or misconfigured here?
>
> I have pasted the screenshot of Storm UI below for reference.
>
> Thanks
> Bijoy
>
>