You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by jerrypeng <gi...@git.apache.org> on 2015/11/04 16:59:26 UTC
[GitHub] storm pull request: [STORM-1165] - normalize the scales of CPU/Mem...
GitHub user jerrypeng opened a pull request:
https://github.com/apache/storm/pull/856
[STORM-1165] - normalize the scales of CPU/Mem/Net when choosing the best node for Resource Aware Scheduler
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerrypeng/storm STORM-1165
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/856.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #856
----
commit 780a472ad2dc4e838b86f65f0c40eb9443046038
Author: Boyang Jerry Peng <je...@yahoo-inc.com>
Date: 2015-11-03T23:13:08Z
[YSTORM-1433] - normalize the scales of CPU/Mem/Net when choosing the best node
commit e638d9a20fac44844f59ee911d3d15bc726d0f5d
Author: Boyang Jerry Peng <je...@yahoo-inc.com>
Date: 2015-11-04T15:57:59Z
add a +1 in the denominator to deal with the case if available resources is zero
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1165] - normalize the scales of CPU/Mem...
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on a diff in the pull request:
https://github.com/apache/storm/pull/856#discussion_r43907626
--- Diff: storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/ResourceAwareStrategy.java ---
@@ -209,18 +209,18 @@ private WorkerSlot getBestWorker(ExecutorDetails exec, TopologyDetails td, Strin
for (RAS_Node n : nodes) {
if(n.getFreeSlots().size()>0) {
if (n.getAvailableMemoryResources() >= taskMem
- && n.getAvailableCpuResources() >= taskCPU) {
- double a = Math.pow((taskCPU - n.getAvailableCpuResources())
- * this.CPU_WEIGHT, 2);
- double b = Math.pow((taskMem - n.getAvailableMemoryResources())
- * this.MEM_WEIGHT, 2);
- double c = 0.0;
- if(this.refNode != null) {
- c = Math.pow(this.distToNode(this.refNode, n)
- * this.NETWORK_WEIGHT, 2);
- }
- double distance = Math.sqrt(a + b + c);
- nodeRankMap.put(distance, n);
+ && n.getAvailableCpuResources() >= taskCPU) {
+ double a = Math.pow(((taskCPU - n.getAvailableCpuResources())/n.getAvailableCpuResources() + 1)
+ * this.CPU_WEIGHT, 2);
+ double b = Math.pow(((taskMem - n.getAvailableMemoryResources())/n.getAvailableMemoryResources() + 1)
--- End diff --
Should this be (n.getAvailableMemoryResources()+1) to avoid "divide by 0"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1165] - normalize the scales of CPU/Mem...
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/856#issuecomment-153786537
+1 looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1165] - normalize the scales of CPU/Mem...
Posted by jerrypeng <gi...@git.apache.org>.
Github user jerrypeng commented on the pull request:
https://github.com/apache/storm/pull/856#issuecomment-153797453
@zhuoliu thanks for your thorough review!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1165] - normalize the scales of CPU/Mem...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/storm/pull/856#discussion_r43921298
--- Diff: storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/ResourceAwareStrategy.java ---
@@ -209,18 +209,18 @@ private WorkerSlot getBestWorker(ExecutorDetails exec, TopologyDetails td, Strin
for (RAS_Node n : nodes) {
if(n.getFreeSlots().size()>0) {
if (n.getAvailableMemoryResources() >= taskMem
- && n.getAvailableCpuResources() >= taskCPU) {
- double a = Math.pow((taskCPU - n.getAvailableCpuResources())
- * this.CPU_WEIGHT, 2);
- double b = Math.pow((taskMem - n.getAvailableMemoryResources())
- * this.MEM_WEIGHT, 2);
- double c = 0.0;
- if(this.refNode != null) {
- c = Math.pow(this.distToNode(this.refNode, n)
- * this.NETWORK_WEIGHT, 2);
- }
- double distance = Math.sqrt(a + b + c);
- nodeRankMap.put(distance, n);
+ && n.getAvailableCpuResources() >= taskCPU) {
+ double a = Math.pow(((taskCPU - n.getAvailableCpuResources())/(n.getAvailableCpuResources() + 1))
+ * this.CPU_WEIGHT, 2);
+ double b = Math.pow(((taskMem - n.getAvailableMemoryResources())/(n.getAvailableMemoryResources() + 1))
+ * this.MEM_WEIGHT, 2);
+ double c = 0.0;
+ if(this.refNode != null) {
+ c = Math.pow(this.distToNode(this.refNode, n)
+ * this.NETWORK_WEIGHT, 2);
+ }
+ double distance = Math.sqrt(a + b + c);
+ nodeRankMap.put(distance, n);
--- End diff --
sorry never mind +1 NB
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1165] - normalize the scales of CPU/Mem...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/storm/pull/856#discussion_r43921033
--- Diff: storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/ResourceAwareStrategy.java ---
@@ -209,18 +209,18 @@ private WorkerSlot getBestWorker(ExecutorDetails exec, TopologyDetails td, Strin
for (RAS_Node n : nodes) {
if(n.getFreeSlots().size()>0) {
if (n.getAvailableMemoryResources() >= taskMem
- && n.getAvailableCpuResources() >= taskCPU) {
- double a = Math.pow((taskCPU - n.getAvailableCpuResources())
- * this.CPU_WEIGHT, 2);
- double b = Math.pow((taskMem - n.getAvailableMemoryResources())
- * this.MEM_WEIGHT, 2);
- double c = 0.0;
- if(this.refNode != null) {
- c = Math.pow(this.distToNode(this.refNode, n)
- * this.NETWORK_WEIGHT, 2);
- }
- double distance = Math.sqrt(a + b + c);
- nodeRankMap.put(distance, n);
+ && n.getAvailableCpuResources() >= taskCPU) {
+ double a = Math.pow(((taskCPU - n.getAvailableCpuResources())/(n.getAvailableCpuResources() + 1))
+ * this.CPU_WEIGHT, 2);
+ double b = Math.pow(((taskMem - n.getAvailableMemoryResources())/(n.getAvailableMemoryResources() + 1))
+ * this.MEM_WEIGHT, 2);
+ double c = 0.0;
+ if(this.refNode != null) {
+ c = Math.pow(this.distToNode(this.refNode, n)
+ * this.NETWORK_WEIGHT, 2);
+ }
+ double distance = Math.sqrt(a + b + c);
+ nodeRankMap.put(distance, n);
--- End diff --
indentation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1165] - normalize the scales of CPU/Mem...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/856
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1165] - normalize the scales of CPU/Mem...
Posted by zhuoliu <gi...@git.apache.org>.
Github user zhuoliu commented on the pull request:
https://github.com/apache/storm/pull/856#issuecomment-153797173
Looks good. +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: [STORM-1165] - normalize the scales of CPU/Mem...
Posted by jerrypeng <gi...@git.apache.org>.
Github user jerrypeng commented on a diff in the pull request:
https://github.com/apache/storm/pull/856#discussion_r43907913
--- Diff: storm-core/src/jvm/backtype/storm/scheduler/resource/strategies/ResourceAwareStrategy.java ---
@@ -209,18 +209,18 @@ private WorkerSlot getBestWorker(ExecutorDetails exec, TopologyDetails td, Strin
for (RAS_Node n : nodes) {
if(n.getFreeSlots().size()>0) {
if (n.getAvailableMemoryResources() >= taskMem
- && n.getAvailableCpuResources() >= taskCPU) {
- double a = Math.pow((taskCPU - n.getAvailableCpuResources())
- * this.CPU_WEIGHT, 2);
- double b = Math.pow((taskMem - n.getAvailableMemoryResources())
- * this.MEM_WEIGHT, 2);
- double c = 0.0;
- if(this.refNode != null) {
- c = Math.pow(this.distToNode(this.refNode, n)
- * this.NETWORK_WEIGHT, 2);
- }
- double distance = Math.sqrt(a + b + c);
- nodeRankMap.put(distance, n);
+ && n.getAvailableCpuResources() >= taskCPU) {
+ double a = Math.pow(((taskCPU - n.getAvailableCpuResources())/n.getAvailableCpuResources() + 1)
+ * this.CPU_WEIGHT, 2);
+ double b = Math.pow(((taskMem - n.getAvailableMemoryResources())/n.getAvailableMemoryResources() + 1)
--- End diff --
Ya you are right forgot the parentheses! good catch!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---