You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Pracheer Agarwal <pr...@gmail.com> on 2017/04/11 03:37:10 UTC
Distribution of workers across cluster with Resource Aware Scheduler
Hi All,
We are in the process of migrating to storm version 1.0.3. The pattern that
we observe with resource aware scheduler is that it launches workers on one
single machine till the time machine is resource wise exhausted. So few of
the machines are fully utilized while few are sitting idle. Can I tune
storm(RAS) to spread the workers across the clusters ? Are there parameters
to control this feature ?
Regards
Pracheer
Re: Fwd: Distribution of workers across cluster with Resource Aware Scheduler
Posted by Jerry Peng <je...@gmail.com>.
Hello Pracheer,
Storm Version 1.1.0 and higher contains a new scheduling strategy for
the Resource Aware Scheduler that will only pack executors from a
single topology as tightly as it can, but it will spread the
scheduling of topologies across all nodes so what you described
shouldn't occur anymore. You can read more about the updated
scheduling strategy at:
http://storm.apache.org/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html
hope this helps!
Best,
Jerry
On Wed, Apr 12, 2017 at 9:12 AM, Bobby Evans
<ev...@yahoo-inc.com.invalid> wrote:
> No there is currently no way to do this with the default strategy. You can write your own resource aware strategy to place the workers differently though. You just need to write a class that implements IStrategy. It can also be selected on a per topology basis.
>
> The main reason for doing this is because it reduces the network load associated with a topology and hopefully speeds it up as a local transfer is much faster than a remote one.
> If you are having problems with it then it might mean that we are not scheduling for the correct resources. One big hole that we are aware of is network. If you need/want network as a resource then please let us know. We should be able to at least implement something simple similar to CPU and Memory, but a fully functional one that takes the complete network topography into account is going to be difficult to do.
>
> - Bobby
>
> On Tuesday, April 11, 2017, 10:32:49 PM CDT, Pracheer Agarwal <pr...@gmail.com> wrote:---------- Forwarded message ----------
> From: Pracheer Agarwal <pr...@gmail.com>
> Date: Tue, Apr 11, 2017 at 9:07 AM
> Subject: Distribution of workers across cluster with Resource Aware
> Scheduler
> To: user@storm.apache.org
>
>
> Hi All,
>
> We are in the process of migrating to storm version 1.0.3. The pattern that
> we observe with resource aware scheduler is that it launches workers on one
> single machine till the time machine is resource wise exhausted. So few of
> the machines are fully utilized while few are sitting idle. Can I tune
> storm(RAS) to spread the workers across the clusters ? Are there parameters
> to control this feature ?
>
> Regards
> Pracheer
Re: Fwd: Distribution of workers across cluster with Resource Aware
Scheduler
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
No there is currently no way to do this with the default strategy. You can write your own resource aware strategy to place the workers differently though. You just need to write a class that implements IStrategy. It can also be selected on a per topology basis.
The main reason for doing this is because it reduces the network load associated with a topology and hopefully speeds it up as a local transfer is much faster than a remote one.
If you are having problems with it then it might mean that we are not scheduling for the correct resources. One big hole that we are aware of is network. If you need/want network as a resource then please let us know. We should be able to at least implement something simple similar to CPU and Memory, but a fully functional one that takes the complete network topography into account is going to be difficult to do.
- Bobby
On Tuesday, April 11, 2017, 10:32:49 PM CDT, Pracheer Agarwal <pr...@gmail.com> wrote:---------- Forwarded message ----------
From: Pracheer Agarwal <pr...@gmail.com>
Date: Tue, Apr 11, 2017 at 9:07 AM
Subject: Distribution of workers across cluster with Resource Aware
Scheduler
To: user@storm.apache.org
Hi All,
We are in the process of migrating to storm version 1.0.3. The pattern that
we observe with resource aware scheduler is that it launches workers on one
single machine till the time machine is resource wise exhausted. So few of
the machines are fully utilized while few are sitting idle. Can I tune
storm(RAS) to spread the workers across the clusters ? Are there parameters
to control this feature ?
Regards
Pracheer
Fwd: Distribution of workers across cluster with Resource Aware Scheduler
Posted by Pracheer Agarwal <pr...@gmail.com>.
---------- Forwarded message ----------
From: Pracheer Agarwal <pr...@gmail.com>
Date: Tue, Apr 11, 2017 at 9:07 AM
Subject: Distribution of workers across cluster with Resource Aware
Scheduler
To: user@storm.apache.org
Hi All,
We are in the process of migrating to storm version 1.0.3. The pattern that
we observe with resource aware scheduler is that it launches workers on one
single machine till the time machine is resource wise exhausted. So few of
the machines are fully utilized while few are sitting idle. Can I tune
storm(RAS) to spread the workers across the clusters ? Are there parameters
to control this feature ?
Regards
Pracheer