You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Spico Florin <sp...@gmail.com> on 2016/02/01 19:15:44 UTC
running multiple topologies in same cluster
Hello!
I have an use case where we have to deploy many tpologies in a storm
cluster.
1.we would like to know if running these topologies in combination with
apache slider over yarn would bring us some benefits in terms of resource
consumption?
2. in such cases (running many topolgies aprox 60) what are the best
practices on how to run them over a cluster with a smart load balanced
hardware resources consumption (cpu, ram)?
I look forward for your answers.
Regards,
Florin
Re: running multiple topologies in same cluster
Posted by Spico Florin <sp...@gmail.com>.
Hello!
Thank you all for your answers! I guess I'll wait for adding in support for
resource aware scheduling in a multi-tenant stand alone storm cluster.
Regards,
Florin
On Tue, Feb 2, 2016 at 1:00 AM, Erik Weathers <ew...@groupon.com> wrote:
> hi Spico,
>
> As Bobby said, native Storm is going to have better support for this soon.
>
> FWIW, there is also the storm-on-mesos project, which we've been running
> on for almost 2 years at Groupon.
>
> - https://github.com/mesos/storm
>
> Caveats:
>
> - storm's logviewer is unsupported
> - scheduling can be suboptimal, causing topologies of different "size"
> (resource requirements) to starve each other
> - side effect of needing to dynamically calculate storm "slots"
> from mesos resource offers, my team has a framework change we are testing
> that will improve that behavior.
> - it's relatively complex to operate since there are so many different
> moving parts / components
> - mesos-master
> - mesos-slave/agent (it's being renamed)
> - storm nimbus (MesosNimbus -> the mesos scheduler)
> - storm supervisor (MesosSupervisor -> the mesos executor)
> - storm worker (the mesos task)
> - ZooKeeper
> - it probably won't work nicely with all the fancy security stuff that
> has been added to Storm in 0.10.0+
>
> - Erik
>
> On Mon, Feb 1, 2016 at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com> wrote:
>
>> We are currently adding in support for resource aware scheduling in a
>> multi-tenant stand alone storm cluster. It is still alpha quality but we
>> plan on getting it into production at Yahoo this quarter. If you can wait
>> that would be the preferred way I see to support your use case.
>>
>> - Bobby
>>
>>
>> On Monday, February 1, 2016 12:16 PM, Spico Florin <sp...@gmail.com>
>> wrote:
>>
>>
>> Hello!
>> I have an use case where we have to deploy many tpologies in a storm
>> cluster.
>> 1.we would like to know if running these topologies in combination with
>> apache slider over yarn would bring us some benefits in terms of resource
>> consumption?
>> 2. in such cases (running many topolgies aprox 60) what are the best
>> practices on how to run them over a cluster with a smart load balanced
>> hardware resources consumption (cpu, ram)?
>> I look forward for your answers.
>> Regards,
>> Florin
>>
>>
>>
>>
>
Re: running multiple topologies in same cluster
Posted by Erik Weathers <ew...@groupon.com>.
hi Spico,
As Bobby said, native Storm is going to have better support for this soon.
FWIW, there is also the storm-on-mesos project, which we've been running on
for almost 2 years at Groupon.
- https://github.com/mesos/storm
Caveats:
- storm's logviewer is unsupported
- scheduling can be suboptimal, causing topologies of different "size"
(resource requirements) to starve each other
- side effect of needing to dynamically calculate storm "slots" from
mesos resource offers, my team has a framework change we are testing that
will improve that behavior.
- it's relatively complex to operate since there are so many different
moving parts / components
- mesos-master
- mesos-slave/agent (it's being renamed)
- storm nimbus (MesosNimbus -> the mesos scheduler)
- storm supervisor (MesosSupervisor -> the mesos executor)
- storm worker (the mesos task)
- ZooKeeper
- it probably won't work nicely with all the fancy security stuff that
has been added to Storm in 0.10.0+
- Erik
On Mon, Feb 1, 2016 at 12:14 PM, Bobby Evans <ev...@yahoo-inc.com> wrote:
> We are currently adding in support for resource aware scheduling in a
> multi-tenant stand alone storm cluster. It is still alpha quality but we
> plan on getting it into production at Yahoo this quarter. If you can wait
> that would be the preferred way I see to support your use case.
>
> - Bobby
>
>
> On Monday, February 1, 2016 12:16 PM, Spico Florin <sp...@gmail.com>
> wrote:
>
>
> Hello!
> I have an use case where we have to deploy many tpologies in a storm
> cluster.
> 1.we would like to know if running these topologies in combination with
> apache slider over yarn would bring us some benefits in terms of resource
> consumption?
> 2. in such cases (running many topolgies aprox 60) what are the best
> practices on how to run them over a cluster with a smart load balanced
> hardware resources consumption (cpu, ram)?
> I look forward for your answers.
> Regards,
> Florin
>
>
>
>
Re: running multiple topologies in same cluster
Posted by Bobby Evans <ev...@yahoo-inc.com>.
We are currently adding in support for resource aware scheduling in a multi-tenant stand alone storm cluster. It is still alpha quality but we plan on getting it into production at Yahoo this quarter. If you can wait that would be the preferred way I see to support your use case.
- Bobby
On Monday, February 1, 2016 12:16 PM, Spico Florin <sp...@gmail.com> wrote:
Hello!
I have an use case where we have to deploy many tpologies in a storm cluster.
1.we would like to know if running these topologies in combination with apache slider over yarn would bring us some benefits in terms of resource consumption?
2. in such cases (running many topolgies aprox 60) what are the best practices on how to run them over a cluster with a smart load balanced hardware resources consumption (cpu, ram)?
I look forward for your answers.
Regards,
Florin