You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Erik Weathers <ew...@groupon.com> on 2015/09/15 00:04:21 UTC
Storm multi-tenancy in 0.10 - is there any topology resource isolation?
hi Storm Devs,
After reading the release notes for 0.10-beta, I was unsure if the
multi-tenancy feature will isolate topologies from one another in terms of
their process resources (CPU, memory):
- https://storm.apache.org/2015/06/15/storm0100-beta-released.html
- Note: there is a reference to: "configurable resource limits."
To understand how this is implemented, I tried searching through the code
(both 0.10.x-branch and master), but couldn't find any reference to
"cgroup". So how is the resource isolation being done?
Maybe this release note is referencing the "R-storm" work? That still
isn't isolating/limiting resources as far as I have been able to discern,
it's instead focused on scheduling. Also it doesn't seem to be fixed in
0.10 already, so I'm doubtful of it being the answer to my question.
- http://web.engr.illinois.edu/~bpeng/files/r-storm.pdf
- https://issues.apache.org/jira/browse/STORM-893
Thanks!
- Erik
Re: Storm multi-tenancy in 0.10 - is there any topology resource
isolation?
Posted by "Boyang(Jerry) Peng" <je...@yahoo-inc.com.INVALID>.
Hello Eric,
I am glad you are interested in my work on Resource Aware Scheduling in Storm. I have a initial implementation of the code on my github:
https://github.com/jerrypeng/storm/tree/opensource_ras
I haven't made a pull request yet but I will make it soon. We are actively improving and adding functionality at Yahoo to the initial work. In the future we will add enforced resource isolation through using CGroups or a something similar.
Best,
Boyang Jerry Peng
On Monday, September 14, 2015 5:05 PM, Erik Weathers <ew...@groupon.com> wrote:
hi Storm Devs,
After reading the release notes for 0.10-beta, I was unsure if the
multi-tenancy feature will isolate topologies from one another in terms of
their process resources (CPU, memory):
- https://storm.apache.org/2015/06/15/storm0100-beta-released.html
- Note: there is a reference to: "configurable resource limits."
To understand how this is implemented, I tried searching through the code
(both 0.10.x-branch and master), but couldn't find any reference to
"cgroup". So how is the resource isolation being done?
Maybe this release note is referencing the "R-storm" work? That still
isn't isolating/limiting resources as far as I have been able to discern,
it's instead focused on scheduling. Also it doesn't seem to be fixed in
0.10 already, so I'm doubtful of it being the answer to my question.
- http://web.engr.illinois.edu/~bpeng/files/r-storm.pdf
- https://issues.apache.org/jira/browse/STORM-893
Thanks!
- Erik
Re: Storm multi-tenancy in 0.10 - is there any topology resource isolation?
Posted by Erik Weathers <ew...@groupon.com>.
>From reading the SECURITY.md file in the storm code, it seems the answer is
that the "resource limits" feature is not what I pictured. The Storm
0.10 "resource
limiting" functionality provided by the MultitenantScheduler
<https://github.com/apache/storm/blob/master/SECURITY.md#multi-tenant-scheduler>
is
similar to the IsolationScheduler of Storm 0.8.2
<https://storm.apache.org/2013/01/11/storm082-released.html>, which allowed
you to isolate the set of hosts that a topology would run on. In Storm 0.10
we will be able to use the MultitenantScheduler to control "the maximum
number of nodes a user is guaranteed to be able to use for their
topologies" (though I'm not 100% sure what that actually means).
On a related note, there is also a feature in Storm 0.10 to limit the
maximum number of worker & executors that *any* topology can have
<https://github.com/apache/storm/blob/master/SECURITY.md#limits>.
Neither of these features provide the process-level resource isolation that
storm-on-mesos <https://github.com/mesos/storm> provides, nor what JStorm
seems to have implemented
<https://github.com/alibaba/jstorm/wiki/Resource-isolation>. As a
maintainer of storm-on-mesos I was curious if the functionality of that
framework had been subsumed into Storm proper -- not yet is the answer!
- Erik
On Mon, Sep 14, 2015 at 3:04 PM, Erik Weathers <ew...@groupon.com>
wrote:
> hi Storm Devs,
>
> After reading the release notes for 0.10-beta, I was unsure if the
> multi-tenancy feature will isolate topologies from one another in terms of
> their process resources (CPU, memory):
>
> - https://storm.apache.org/2015/06/15/storm0100-beta-released.html
> - Note: there is a reference to: "configurable resource limits."
>
> To understand how this is implemented, I tried searching through the code
> (both 0.10.x-branch and master), but couldn't find any reference to
> "cgroup". So how is the resource isolation being done?
>
> Maybe this release note is referencing the "R-storm" work? That still
> isn't isolating/limiting resources as far as I have been able to discern,
> it's instead focused on scheduling. Also it doesn't seem to be fixed in
> 0.10 already, so I'm doubtful of it being the answer to my question.
>
> - http://web.engr.illinois.edu/~bpeng/files/r-storm.pdf
> - https://issues.apache.org/jira/browse/STORM-893
>
> Thanks!
>
> - Erik
>