You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Arthur Maciejewicz <ar...@signafire.com> on 2016/11/07 16:31:37 UTC
Explicit Topology Parallelism Shape
Hello All,
Is there a way to make the "shape" of topology parallelism explicit?
For example, assume:
* W workers on W nodes
* 2 spouts with 1 executor each (total of 2 executors)
* 1 "mapper" bolt with M executors
* 1 "process" bolt with N executors.
Is it currently possibly to co-locate the spouts with the "mappers" in 2
workers, while pinning the N "process" executors to the remaining (W-2)
workers?
Visually this is what I want to do:
2 Workers W-2 Workers
+-----------+
+----------+ | process |
| spout | +-----------+
| + +--> +-----------+
| map | | process |
+----------+ +-----------+
+-----------+
+----------+ | process |
| spout | +-----------+
| + +--> ...
| map | +-----------+
+----------+ | process |
+-----------+
+-----------+
| process |
+-----------+
Thanks,
Arthur
Re: Explicit Topology Parallelism Shape
Posted by Arthur Maciejewicz <ar...@signafire.com>.
That's unfortunate, I was hoping for a built-in way to do it, but that's
ok. Thanks for the quick response.
- Arthur
On Mon, Nov 7, 2016 at 11:56 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
> Yes, to do what you want you would need to implement a custom scheduler.
>
> More details can be found here: http://storm.apache.org/
> releases/1.0.2/Storm-Scheduler.html
>
> -Taylor
>
> On Nov 7, 2016, at 11:31 AM, Arthur Maciejewicz <ar...@signafire.com>
> wrote:
>
> Hello All,
>
> Is there a way to make the "shape" of topology parallelism explicit?
> For example, assume:
>
> * W workers on W nodes
> * 2 spouts with 1 executor each (total of 2 executors)
> * 1 "mapper" bolt with M executors
> * 1 "process" bolt with N executors.
>
> Is it currently possibly to co-locate the spouts with the "mappers" in 2
> workers, while pinning the N "process" executors to the remaining (W-2)
> workers?
>
> Visually this is what I want to do:
>
>
> 2 Workers W-2 Workers
>
> +-----------+
> +----------+ | process |
> | spout | +-----------+
> | + +--> +-----------+
> | map | | process |
> +----------+ +-----------+
> +-----------+
> +----------+ | process |
> | spout | +-----------+
> | + +--> ...
> | map | +-----------+
> +----------+ | process |
> +-----------+
> +-----------+
> | process |
> +-----------+
>
> Thanks,
>
> Arthur
>
>
>
Re: Explicit Topology Parallelism Shape
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Yes, to do what you want you would need to implement a custom scheduler.
More details can be found here: http://storm.apache.org/releases/1.0.2/Storm-Scheduler.html <http://storm.apache.org/releases/1.0.2/Storm-Scheduler.html>
-Taylor
> On Nov 7, 2016, at 11:31 AM, Arthur Maciejewicz <ar...@signafire.com> wrote:
>
> Hello All,
>
> Is there a way to make the "shape" of topology parallelism explicit? For example, assume:
>
> * W workers on W nodes
> * 2 spouts with 1 executor each (total of 2 executors)
> * 1 "mapper" bolt with M executors
> * 1 "process" bolt with N executors.
>
> Is it currently possibly to co-locate the spouts with the "mappers" in 2 workers, while pinning the N "process" executors to the remaining (W-2) workers?
>
> Visually this is what I want to do:
>
>
> 2 Workers W-2 Workers
>
> +-----------+
> +----------+ | process |
> | spout | +-----------+
> | + +--> +-----------+
> | map | | process |
> +----------+ +-----------+
> +-----------+
> +----------+ | process |
> | spout | +-----------+
> | + +--> ...
> | map | +-----------+
> +----------+ | process |
> +-----------+
> +-----------+
> | process |
> +-----------+
>
> Thanks,
>
> Arthur