You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by gyfora <gi...@git.apache.org> on 2016/06/18 10:40:24 UTC
[GitHub] flink pull request #2129: [FLINK-1003] [WIP] Spread out scheduling of tasks
GitHub user gyfora opened a pull request:
https://github.com/apache/flink/pull/2129
[FLINK-1003] [WIP] Spread out scheduling of tasks
This is a working progress PR with the core functionality implemented but no tests yet.
As this is a highly critical part of the system I would like to get some initial feedback before proceeding to write / change a huge amount of tests :)
About the functionality:
This is an adaptation of https://github.com/apache/flink/pull/60 to the current flink scheduler. Instead of preferring local instances when scheduling new task slots the new scheduling strategy allows users to balance the load on the different task managers.
Every time a new task needs to be scheduled the scheduler considers all instances that satisfy the scheduling constraints (has available nodes + locality constraints) and picks the one with the smallest load. Load is calculated by the percentage of task slots occupied in a given task manager.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gyfora/flink scheduling
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2129.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2129
----
commit f895fd71f392482cf0a50e32dc637f7885995c4e
Author: Gyula Fora <gy...@apache.org>
Date: 2016-06-18T10:19:13Z
[FLINK-1003] Spread out scheduling of tasks
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2129: [FLINK-1003] [WIP] Spread out scheduling of tasks
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the issue:
https://github.com/apache/flink/pull/2129
I think this is superseded with the current work on dynamic scaling as part of FLIP-6. Can we close this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2129: [FLINK-1003] [WIP] Spread out scheduling of tasks
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the issue:
https://github.com/apache/flink/pull/2129
Thanks Stephan,
I agree that there is a lot to think about/improve when it comes to scheduling and dynamic scaling. Should we add this to the Key groups design doc or there is going to be a bigger one more general about different system aspects of the dynamic scaling?
It might make sense to introduce some sort of a scheduling hint for when we redeploy a job with higher parallelism so that we can exploit some state checkpoint locality for quicker recovery, but this is another topic.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2129: [FLINK-1003] [WIP] Spread out scheduling of tasks
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora closed the pull request at:
https://github.com/apache/flink/pull/2129
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2129: [FLINK-1003] [WIP] Spread out scheduling of tasks
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2129
I think the implementation looks good.
The big question for this is actually more whether this interplays nicely with other changes around dynamic scaling. Once we have that part figured out and we see it works together well, this change is probably good.
In order to figure that out, let's try and actually collect and write up the designs for how the system will evolve with respect to dynamic scaling.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---