storm scheduler behavior

[Weide Zhang <we...@gmail.com>]

Hi

Is there any good documentation how storm does resource allocation and how
does it maps a topology to different work process and how was it decided to
group components in one worker process or across worker process ?

i'm trying to understand what algorithm it uses to partition the topology
and in the meanwhile considering the resource constraint to achieve best
performance if there is something available here ?

Thanks,

Weide

Re: storm scheduler behavior

[Srinath C <sr...@gmail.com>]

Hi Weide,

I haven't found any material on this matter. But as far as I could figure
out, the strategy seems to try to evenly divide the total number of
executors among the worker processes. If a particular spout/bolt has
multiple executors, it tries to span them across all the worker processes.

However, I could not find out if the topology layout or groupings between
the components have any role to play in the distribution.

Regards,
Srinath.



On Sun, May 11, 2014 at 10:12 PM, Weide Zhang <we...@gmail.com> wrote:

> Hi
>
> Is there any good documentation how storm does resource allocation and how
> does it maps a topology to different work process and how was it decided to
> group components in one worker process or across worker process ?
>
> i'm trying to understand what algorithm it uses to partition the topology
> and in the meanwhile considering the resource constraint to achieve best
> performance if there is something available here ?
>
> Thanks,
>
> Weide
>

Re: storm scheduler behavior

["M.Tarkeshwar Rao" <ta...@gmail.com>]

Hi,

I am facing a problem in scheduling. we are using trident.
Initially co-ordinator and emiter scheduled in one worker machine and other
bolts in different machines of the cluster.

When Master-Batch-controller of trident goes down or intentionally killed,
nimbus bring the MBC up but we are getting so many failures in Storm ui.

without scheduling it is working fine but with scheduling we are getting
failures.

Regards
Tarkesh


On Mon, May 12, 2014 at 8:56 AM, 904689262@qq.com <90...@qq.com> wrote:

> Hi Weide
> there is one paper about Scheduler which used in Storm, and they propose a
> new algorithm to adjust the default assignment.
> http://dl.acm.org/citation.cfm?id=2488267  Adaptive online scheduling in
> storm
>
> ------------------------------
> 904689262@qq.com
>
>
> *From:* Weide Zhang <we...@gmail.com>
> *Date:* 2014-05-12 00:42
> *To:* user <us...@storm.incubator.apache.org>
> *Subject:* storm scheduler behavior
> Hi
>
> Is there any good documentation how storm does resource allocation and how
> does it maps a topology to different work process and how was it decided to
> group components in one worker process or across worker process ?
>
> i'm trying to understand what algorithm it uses to partition the topology
> and in the meanwhile considering the resource constraint to achieve best
> performance if there is something available here ?
>
> Thanks,
>
> Weide
>
>

Re: storm scheduler behavior

["904689262@qq.com" <90...@qq.com>]

Hi

Is there any good documentation how storm does resource allocation and how
does it maps a topology to different work process and how was it decided to
group components in one worker process or across worker process ?

i'm trying to understand what algorithm it uses to partition the topology
and in the meanwhile considering the resource constraint to achieve best
performance if there is something available here ?

Thanks,

Weide