You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Nick R. Katsipoulakis" <ni...@gmail.com> on 2015/02/23 19:56:30 UTC
Manage parallelism in Storm
Hello,
I am trying to control my cluster's parallelism and I want to achieve the
following: 1 worker per component, and 1 executioner per worker and 1 task
per executioner. In order to achieve the former, I submit my components to
the topology as follows:
Config conf = new Config();
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout_1",
new GreenSpout(), 1)
.setNumTasks(1);
builder.setBolt("bolt_1",
new GreenBolt(), 1)
.setNumTasks(1)
.directGrouping("spout_1");
builder.setBolt("bolt_2",
new RedBolt(), 1)
.setNumTasks(1)
.directGrouping("bolt_1");
builder.setBolt("bolt_3",
new YellowBolt(), 1)
.setNumTasks(1)
.directGrouping("bolt_2");
conf.setDebug(true);
conf.setNumWorkers(4);
StormSubmitter.submitTopology("experimental-top", conf,
builder.createTopology());
The above not only sets the parallelism hint (executioner threads per
worker) to 1, but also the number of workers per component to 1. However,
when I submit my topology, and I perform a storm list, I can see that I
have 4 workers and 8 tasks running. Is this normal? My goal is to have each
component executed by 1 thread on each machine. Am I missing something?
Thanks,
Nick
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
Re: Manage parallelism in Storm
Posted by Nathan Leung <nc...@gmail.com>.
The docs have a good discussion of how acks work:
https://storm.apache.org/documentation/Guaranteeing-message-processing.html
On Mon, Feb 23, 2015 at 2:14 PM, Nick R. Katsipoulakis <
nick.katsip@gmail.com> wrote:
> Ok, that makes sense. However, I do not understand how ackers work. Are
> they spouts/bolts exactly the same as my original spouts/bolts and they
> perform different functions? Also, is there any way in my code to
> differentiate the behavior of a spout/bolt with its acker?
>
> Thanks,
> Nick
>
> 2015-02-23 14:09 GMT-05:00 Nathan Leung <nc...@gmail.com>:
>
> with 4 workers, by default, you would also have 4 acker tasks (1 /
>> worker), which is why you might see 8 tasks.
>>
>> On Mon, Feb 23, 2015 at 1:56 PM, Nick R. Katsipoulakis <
>> nick.katsip@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I am trying to control my cluster's parallelism and I want to achieve
>>> the following: 1 worker per component, and 1 executioner per worker and 1
>>> task per executioner. In order to achieve the former, I submit my
>>> components to the topology as follows:
>>>
>>> Config conf = new Config();
>>>
>>> TopologyBuilder builder = new TopologyBuilder();
>>> builder.setSpout("spout_1",
>>> new GreenSpout(), 1)
>>> .setNumTasks(1);
>>> builder.setBolt("bolt_1",
>>> new GreenBolt(), 1)
>>> .setNumTasks(1)
>>> .directGrouping("spout_1");
>>> builder.setBolt("bolt_2",
>>> new RedBolt(), 1)
>>> .setNumTasks(1)
>>> .directGrouping("bolt_1");
>>> builder.setBolt("bolt_3",
>>> new YellowBolt(), 1)
>>> .setNumTasks(1)
>>> .directGrouping("bolt_2");
>>>
>>> conf.setDebug(true);
>>> conf.setNumWorkers(4);
>>> StormSubmitter.submitTopology("experimental-top", conf,
>>> builder.createTopology());
>>>
>>> The above not only sets the parallelism hint (executioner threads per
>>> worker) to 1, but also the number of workers per component to 1. However,
>>> when I submit my topology, and I perform a storm list, I can see that I
>>> have 4 workers and 8 tasks running. Is this normal? My goal is to have each
>>> component executed by 1 thread on each machine. Am I missing something?
>>>
>>> Thanks,
>>> Nick
>>>
>>>
>>> --
>>> Nikolaos Romanos Katsipoulakis,
>>> University of Pittsburgh, PhD candidate
>>>
>>
>>
>
>
> --
> Nikolaos Romanos Katsipoulakis,
> University of Pittsburgh, PhD candidate
>
Re: Manage parallelism in Storm
Posted by "Nick R. Katsipoulakis" <ni...@gmail.com>.
Ok, that makes sense. However, I do not understand how ackers work. Are
they spouts/bolts exactly the same as my original spouts/bolts and they
perform different functions? Also, is there any way in my code to
differentiate the behavior of a spout/bolt with its acker?
Thanks,
Nick
2015-02-23 14:09 GMT-05:00 Nathan Leung <nc...@gmail.com>:
> with 4 workers, by default, you would also have 4 acker tasks (1 /
> worker), which is why you might see 8 tasks.
>
> On Mon, Feb 23, 2015 at 1:56 PM, Nick R. Katsipoulakis <
> nick.katsip@gmail.com> wrote:
>
>> Hello,
>>
>> I am trying to control my cluster's parallelism and I want to achieve the
>> following: 1 worker per component, and 1 executioner per worker and 1 task
>> per executioner. In order to achieve the former, I submit my components to
>> the topology as follows:
>>
>> Config conf = new Config();
>>
>> TopologyBuilder builder = new TopologyBuilder();
>> builder.setSpout("spout_1",
>> new GreenSpout(), 1)
>> .setNumTasks(1);
>> builder.setBolt("bolt_1",
>> new GreenBolt(), 1)
>> .setNumTasks(1)
>> .directGrouping("spout_1");
>> builder.setBolt("bolt_2",
>> new RedBolt(), 1)
>> .setNumTasks(1)
>> .directGrouping("bolt_1");
>> builder.setBolt("bolt_3",
>> new YellowBolt(), 1)
>> .setNumTasks(1)
>> .directGrouping("bolt_2");
>>
>> conf.setDebug(true);
>> conf.setNumWorkers(4);
>> StormSubmitter.submitTopology("experimental-top", conf,
>> builder.createTopology());
>>
>> The above not only sets the parallelism hint (executioner threads per
>> worker) to 1, but also the number of workers per component to 1. However,
>> when I submit my topology, and I perform a storm list, I can see that I
>> have 4 workers and 8 tasks running. Is this normal? My goal is to have each
>> component executed by 1 thread on each machine. Am I missing something?
>>
>> Thanks,
>> Nick
>>
>>
>> --
>> Nikolaos Romanos Katsipoulakis,
>> University of Pittsburgh, PhD candidate
>>
>
>
--
Nikolaos Romanos Katsipoulakis,
University of Pittsburgh, PhD candidate
Re: Manage parallelism in Storm
Posted by Nathan Leung <nc...@gmail.com>.
with 4 workers, by default, you would also have 4 acker tasks (1 / worker),
which is why you might see 8 tasks.
On Mon, Feb 23, 2015 at 1:56 PM, Nick R. Katsipoulakis <
nick.katsip@gmail.com> wrote:
> Hello,
>
> I am trying to control my cluster's parallelism and I want to achieve the
> following: 1 worker per component, and 1 executioner per worker and 1 task
> per executioner. In order to achieve the former, I submit my components to
> the topology as follows:
>
> Config conf = new Config();
>
> TopologyBuilder builder = new TopologyBuilder();
> builder.setSpout("spout_1",
> new GreenSpout(), 1)
> .setNumTasks(1);
> builder.setBolt("bolt_1",
> new GreenBolt(), 1)
> .setNumTasks(1)
> .directGrouping("spout_1");
> builder.setBolt("bolt_2",
> new RedBolt(), 1)
> .setNumTasks(1)
> .directGrouping("bolt_1");
> builder.setBolt("bolt_3",
> new YellowBolt(), 1)
> .setNumTasks(1)
> .directGrouping("bolt_2");
>
> conf.setDebug(true);
> conf.setNumWorkers(4);
> StormSubmitter.submitTopology("experimental-top", conf,
> builder.createTopology());
>
> The above not only sets the parallelism hint (executioner threads per
> worker) to 1, but also the number of workers per component to 1. However,
> when I submit my topology, and I perform a storm list, I can see that I
> have 4 workers and 8 tasks running. Is this normal? My goal is to have each
> component executed by 1 thread on each machine. Am I missing something?
>
> Thanks,
> Nick
>
>
> --
> Nikolaos Romanos Katsipoulakis,
> University of Pittsburgh, PhD candidate
>