You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tillrohrmann <gi...@git.apache.org> on 2014/07/06 20:18:36 UTC
[GitHub] incubator-flink pull request: [FLINK-1003] Added spread out schedu...
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/incubator-flink/pull/60
[FLINK-1003] Added spread out scheduling strategy
The spread out scheduling strategy tries to keep the work load among the instances balanced. The scheduling strategy can be specified by the configuration parameter ```jobmanager.scheduling-strategy```. Valid options are ```SpreadOut``` and ```FillFirst```. The latter strategy is the old and default strategy.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/incubator-flink FLINK-1003
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/60.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #60
----
commit 7800b3becdf3551e69a2f950412476cf65b8534f
Author: Till Rohrmann <ti...@gmail.com>
Date: 2014-07-06T17:24:10Z
Added spread out scheduling strategy
commit e8b573d9721eb78477e154bc050dabb015a0fade
Author: Till Rohrmann <ti...@gmail.com>
Date: 2014-07-06T18:16:06Z
minor fixes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1003] Added spread out schedu...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/incubator-flink/pull/60#issuecomment-62580255
It is not included in FLINK-1030. This implementation is incompatible with the current scheduler implementation, unfortunately.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1003] Added spread out schedu...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:
https://github.com/apache/incubator-flink/pull/60#issuecomment-48249942
Yeah you're right Ufuk, test cases are definitely missing. I try to add them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1003] Added spread out schedu...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/60#discussion_r14598723
--- Diff: stratosphere-tests/src/test/java/eu/stratosphere/test/exampleJavaPrograms/WordCountITCase.java ---
@@ -26,7 +27,8 @@
public WordCountITCase(){
setDegreeOfParallelism(4);
setNumTaskTracker(2);
- setTaskManagerNumSlots(2);
+ setTaskManagerNumSlots(4);
--- End diff --
No you are right, but I wanted to verify that the the tasks are spread out across the two task managers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1003] Added spread out schedu...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/incubator-flink/pull/60#issuecomment-62558527
@StephanEwen Did you include this PR in FLINK-1030 or is it still up for discussion / merging?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1003] Added spread out schedu...
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/incubator-flink/pull/60#issuecomment-62693849
Is it worth keeping this PR open and adapting it or is it easier to start from scratch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1003] Added spread out schedu...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/incubator-flink/pull/60#discussion_r14594319
--- Diff: stratosphere-tests/src/test/java/eu/stratosphere/test/exampleJavaPrograms/WordCountITCase.java ---
@@ -26,7 +27,8 @@
public WordCountITCase(){
setDegreeOfParallelism(4);
setNumTaskTracker(2);
- setTaskManagerNumSlots(2);
+ setTaskManagerNumSlots(4);
--- End diff --
With a dop of 4, it should be enough to have 2 taskmanagers with 2 slots. Or is the number of slots the total number?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1003] Added spread out schedu...
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the pull request:
https://github.com/apache/incubator-flink/pull/60#issuecomment-48191640
Great! :-) Looks good to me minus the following two comments:
1. I don't have a better idea right now, but the SchedulingStrategy name `FillFirst` might be confusing as all instances are filled and not just the the first -- which someone might think by reading the name.
2. I think we should add some test cases before we merge this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1003] Added spread out schedu...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/incubator-flink/pull/60#issuecomment-54075422
This is a very nice idea. I'll try and incorporate it into the following effort: https://issues.apache.org/jira/browse/FLINK-1030
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---