You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by revans2 <gi...@git.apache.org> on 2015/01/29 21:15:11 UTC
[GitHub] storm pull request: STORM-162: Load Balancing Shuffle Grouping
GitHub user revans2 opened a pull request:
https://github.com/apache/storm/pull/402
STORM-162: Load Balancing Shuffle Grouping
This pull request adds a notion of load that is measured and sent to upstream workers through the messaging system. Shuffle grouping was then updated so that if it is configured to do so it will take the load into account.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/revans2/incubator-storm STORM-162
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/402.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #402
----
commit e2a8a038a185bd65b58ac66b61cc0eb926e72e74
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Date: 2015-01-29T20:10:16Z
STORM-162: Load Balancing Shuffle Grouping
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-162: Load Balancing Shuffle Grouping
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/402#issuecomment-112443293
I have some more things to do. There was a suggestion on the JIRA about trying to incorporate process latency along with the queue length for better results, and I would also like to look at adding in support for network distance to the routing too. So I am closing this for now, and will either put up a new pull request or reopen this one when I have the code ready.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-162: Load Balancing Shuffle Grouping
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on a diff in the pull request:
https://github.com/apache/storm/pull/402#discussion_r24359616
--- Diff: storm-core/src/jvm/backtype/storm/Config.java ---
@@ -199,6 +199,12 @@
public static final Object TOPOLOGY_TUPLE_SERIALIZER_SCHEMA = String.class;
/**
+ * Disable load aware grouping support.
+ */
+ public static final String TOPOLOGY_DISABLE_LOADAWARE = "topology.disable.loadaware";
--- End diff --
Could we call this something like `TOPOLOGY_DISABLE_LOADAWARE_MESSAGING` so it is clear what ceases to be aware of load?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-162: Load Balancing Shuffle Grouping
Posted by d2r <gi...@git.apache.org>.
Github user d2r commented on the pull request:
https://github.com/apache/storm/pull/402#issuecomment-112221495
@revans2 can you upmerge?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-162: Load Balancing Shuffle Grouping
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/402#discussion_r24359884
--- Diff: storm-core/src/jvm/backtype/storm/Config.java ---
@@ -199,6 +199,12 @@
public static final Object TOPOLOGY_TUPLE_SERIALIZER_SCHEMA = String.class;
/**
+ * Disable load aware grouping support.
+ */
+ public static final String TOPOLOGY_DISABLE_LOADAWARE = "topology.disable.loadaware";
--- End diff --
Great idea, I couldn't really come up with a decent description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-162: Load Balancing Shuffle Grouping
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 closed the pull request at:
https://github.com/apache/storm/pull/402
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: STORM-162: Load Balancing Shuffle Grouping
Posted by Luckyfang <gi...@git.apache.org>.
Github user Luckyfang commented on the pull request:
https://github.com/apache/storm/pull/402#issuecomment-153572667
test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---