You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Justin Workman <ju...@gmail.com> on 2014/06/06 18:17:14 UTC
Implications of running multiple topologies without isolation
I am trying to understand the implications of running multiple topologies
on a single cluster without using the isolation scheduler. The way this
appears to work, is isolation at the machine level and not the worker
level.
Our issue right now is that we only have 5 machines to work with. We have
enough resources to run multiple workers per machine, but do not feel
comfortable running each topology on fewer than all 5 machines.
So the main questions are, 1) what issues do I risk running into if I run
multiple topologies on a single cluster with out the isolation scheduler,
or 2) is there a way to isolate at the worker level, ie; each worker
handles tasks for a single topology?
Thanks
Justin
Re: Implications of running multiple topologies without isolation
Posted by Nathan Leung <nc...@gmail.com>.
1) a worker can spawn any number of threads, so you can possibly run into
standard shared resources issues (CPU, network, disk, etc). RAM is not as
big of a problem since each worker gets a fixed amount.
2) a worker is spawned for a particular topology; it only execute
spout/bolt tasks for the topology to which it is assigned.
On Fri, Jun 6, 2014 at 12:17 PM, Justin Workman <ju...@gmail.com>
wrote:
> I am trying to understand the implications of running multiple topologies
> on a single cluster without using the isolation scheduler. The way this
> appears to work, is isolation at the machine level and not the worker
> level.
>
> Our issue right now is that we only have 5 machines to work with. We have
> enough resources to run multiple workers per machine, but do not feel
> comfortable running each topology on fewer than all 5 machines.
>
> So the main questions are, 1) what issues do I risk running into if I run
> multiple topologies on a single cluster with out the isolation scheduler,
> or 2) is there a way to isolate at the worker level, ie; each worker
> handles tasks for a single topology?
>
> Thanks
> Justin
>
Re: Implications of running multiple topologies without isolation
Posted by Lin Zhao <li...@groupon.com>.
Try storm on mesos for isolation.
http://mesosphere.io/learn/run-storm-on-mesos/
On Fri, Jun 6, 2014 at 10:03 AM, Justin Workman <ju...@gmail.com>
wrote:
> Thanks for the responses. I assumed worker isolation worked this way.
> I had just read a couple things that made me question this.
>
> Justin
>
> Sent from my iPhone
>
> On Jun 6, 2014, at 10:25 AM, Derek Dagit <de...@yahoo-inc.com> wrote:
>
> >> So the main questions are, 1) what issues do I risk running into if I
> run
> >> multiple topologies on a single cluster with out the isolation
> scheduler,
> >
> > Resource contention on shared boxes: CPU (#cores), network, disk (if
> applicable).
> >
> > Depends on what these topologies are doing: which resources they will
> use the most.
> >
> >
> >> 2) is there a way to isolate at the worker level, ie; each worker
> >> handles tasks for a single topology?
> >
> > I thought this is the way it normally worked. A single worker JVM would
> run on behalf of one topology, but could run tasks from multiple different
> components (bolt/spouts) defined in that topology.
> >
> > --
> > Derek
> >
> >> On 6/6/14, 11:17, Justin Workman wrote:
> >> I am trying to understand the implications of running multiple
> topologies
> >> on a single cluster without using the isolation scheduler. The way this
> >> appears to work, is isolation at the machine level and not the worker
> >> level.
> >>
> >> Our issue right now is that we only have 5 machines to work with. We
> have
> >> enough resources to run multiple workers per machine, but do not feel
> >> comfortable running each topology on fewer than all 5 machines.
> >>
> >> So the main questions are, 1) what issues do I risk running into if I
> run
> >> multiple topologies on a single cluster with out the isolation
> scheduler,
> >> or 2) is there a way to isolate at the worker level, ie; each worker
> >> handles tasks for a single topology?
> >>
> >> Thanks
> >> Justin
> >>
>
--
Lin Zhao
3101 Park Blvd, Palo Alto, CA 94306
Re: Implications of running multiple topologies without isolation
Posted by Justin Workman <ju...@gmail.com>.
Thanks for the responses. I assumed worker isolation worked this way.
I had just read a couple things that made me question this.
Justin
Sent from my iPhone
On Jun 6, 2014, at 10:25 AM, Derek Dagit <de...@yahoo-inc.com> wrote:
>> So the main questions are, 1) what issues do I risk running into if I run
>> multiple topologies on a single cluster with out the isolation scheduler,
>
> Resource contention on shared boxes: CPU (#cores), network, disk (if applicable).
>
> Depends on what these topologies are doing: which resources they will use the most.
>
>
>> 2) is there a way to isolate at the worker level, ie; each worker
>> handles tasks for a single topology?
>
> I thought this is the way it normally worked. A single worker JVM would run on behalf of one topology, but could run tasks from multiple different components (bolt/spouts) defined in that topology.
>
> --
> Derek
>
>> On 6/6/14, 11:17, Justin Workman wrote:
>> I am trying to understand the implications of running multiple topologies
>> on a single cluster without using the isolation scheduler. The way this
>> appears to work, is isolation at the machine level and not the worker
>> level.
>>
>> Our issue right now is that we only have 5 machines to work with. We have
>> enough resources to run multiple workers per machine, but do not feel
>> comfortable running each topology on fewer than all 5 machines.
>>
>> So the main questions are, 1) what issues do I risk running into if I run
>> multiple topologies on a single cluster with out the isolation scheduler,
>> or 2) is there a way to isolate at the worker level, ie; each worker
>> handles tasks for a single topology?
>>
>> Thanks
>> Justin
>>
Re: Implications of running multiple topologies without isolation
Posted by Derek Dagit <de...@yahoo-inc.com>.
> So the main questions are, 1) what issues do I risk running into if I run
> multiple topologies on a single cluster with out the isolation scheduler,
Resource contention on shared boxes: CPU (#cores), network, disk (if applicable).
Depends on what these topologies are doing: which resources they will use the most.
>2) is there a way to isolate at the worker level, ie; each worker
> handles tasks for a single topology?
I thought this is the way it normally worked. A single worker JVM would run on behalf of one topology, but could run tasks from multiple different components (bolt/spouts) defined in that topology.
--
Derek
On 6/6/14, 11:17, Justin Workman wrote:
> I am trying to understand the implications of running multiple topologies
> on a single cluster without using the isolation scheduler. The way this
> appears to work, is isolation at the machine level and not the worker
> level.
>
> Our issue right now is that we only have 5 machines to work with. We have
> enough resources to run multiple workers per machine, but do not feel
> comfortable running each topology on fewer than all 5 machines.
>
> So the main questions are, 1) what issues do I risk running into if I run
> multiple topologies on a single cluster with out the isolation scheduler,
> or 2) is there a way to isolate at the worker level, ie; each worker
> handles tasks for a single topology?
>
> Thanks
> Justin
>