You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2015/10/08 23:21:26 UTC
[jira] [Commented] (STORM-1093) Launching Workers with resources
specified in resource-aware schedulers
[ https://issues.apache.org/jira/browse/STORM-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949416#comment-14949416 ]
ASF GitHub Bot commented on STORM-1093:
---------------------------------------
GitHub user zhuoliu opened a pull request:
https://github.com/apache/storm/pull/790
[STORM-1093] Launching Workers with resources specified in resource-aware schedulers
This request provides a set of schemes that allow nimbus to put different types of resources to each worker slot, then push the resources within assignment to ZooKeepers; also, at the supervisor side, such resources in each worker slot should be used by supervisor for launching a worker's JVM (initially, the JVM heap size, this can also be called by CGroup or Docker for resource segregation).
Such scheme can be used not only by RAS scheduler (STORM-893), but also by any customized scheduler for conducting mem/cpu resource aware scheduling.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhuoliu/storm 1093a
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/790.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #790
----
commit 066fd7aa89a7f50a1f5794a14a68312b5996a112
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-10-07T21:08:36Z
Add resource setting API in WorkerSlot
commit 8b327c9ed84ec2b29ad7c807f04b9928f6d25cbc
Author: Kishor Patil <kp...@yahoo-inc.com>
Date: 2015-08-21T16:37:19Z
Merge pull request #509 from zhuol/1779
[YSTORM-1779] Send/receive resource information from nimbus to supervisor for RAS
commit ca8bb24dbffd806ed41e27976552e197608d6627
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-10-07T21:48:33Z
Remove generated files
commit 715e64ba05bd691555d53d12c0daa851868fea03
Author: Derek Dagit <de...@yahoo-inc.com>
Date: 2015-08-31T16:55:36Z
Cherry-pick YSTORM-1546
commit 3a045bf0f5077df155a00dc11ad859e6b9d6578d
Author: Zhuo Liu <zh...@yahoo-inc.com>
Date: 2015-09-16T22:02:42Z
Cherry-pick YSTORM-2146
commit 545df27ddd79d2a1175749887c1681abfbd74635
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-10-08T15:50:27Z
Minor Fix of the import
commit a684962022ad228397f86c09cb266a56afb75a59
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-10-08T21:19:49Z
Push Thrift file changes
----
> Launching Workers with resources specified in resource-aware schedulers
> -----------------------------------------------------------------------
>
> Key: STORM-1093
> URL: https://issues.apache.org/jira/browse/STORM-1093
> Project: Apache Storm
> Issue Type: Story
> Components: storm-core
> Reporter: Zhuo Liu
> Assignee: Zhuo Liu
>
> Currently, we have Resource Aware Scheduler (STORM-894) in nimbus, which can allocate different types of resource (CPU, onheap-memory, offheap-memory) to the workers assigned to each topology's tasks.
> However, such resources are not visible to the supervisor, therefore, supervisor will still launch workers with fixed amount of heap size memory (e.g., -Xmx=768M).
> Therefore, we need a whole set of schemes that allow nimbus to put different types of resources to each worker slot, then push the resources with assignment to ZooKeepers; also, at the supervisor side, such resources in each worker slot should be used by supervisor for launching a worker's JVM (initially, the JVM heap size).
> This scheme can be used not only by RAS scheduler (STORM-893), but also by any customized scheduler for conducting mem/cpu/network resource specified-scheduling.
> In the future, the resources of memory, CPU and network can also be used by supervisor to launch a worker in a resource-segregated container, such as a CGroup or Docker, with isolated Memory/CPU/Network resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)