You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by wangzhijiang999 <gi...@git.apache.org> on 2017/01/16 09:30:46 UTC
[GitHub] flink pull request #3125: [FLINK-5499][JobManager]Reuse the resource locatio...
GitHub user wangzhijiang999 opened a pull request:
https://github.com/apache/flink/pull/3125
[FLINK-5499][JobManager]Reuse the resource location of prior executio\u2026
Currently when schedule execution to request to allocate slot from **SlotPool**, the **TaskManagerLocation** parameter is empty collection. So for task fail over scenario, the new execution attempt may be deployed to different task managers. If setting rockDB as state backend, the performance is better if the data can be restored from local machine. So we try to reuse the **TaskManagerLocation** of prior execution attempt when allocating slot from **SlotPool**. If the **TaskManagerLocation** is empty from prior executions, the behavior is the same with current status.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangzhijiang999/flink FLINK-5499
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3125.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3125
----
commit ab2e24ae7e82be45359f249670f72664226ec18c
Author: \u6dd8\u6c5f <ta...@alibaba-inc.com>
Date: 2017-01-16T09:28:19Z
[FLINK-5499][JobManager]Reuse the resource location of prior execution attempt in allocating slot
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3125: [FLINK-5499][JobManager]Reuse the resource locatio...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/3125
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3125: [FLINK-5499][JobManager]Reuse the resource location of pr...
Posted by wangzhijiang999 <gi...@git.apache.org>.
Github user wangzhijiang999 commented on the issue:
https://github.com/apache/flink/pull/3125
@StephanEwen
Yes, the current concern is only focusing on state restore performance. This PR does not consider all the scenarios and it may be only the first step for the slot location implementation.
If the location do not exist, it can add other strategies to decide the locations, such as co-loated by input for batch job as you mentioned. And it can be the second step for the implementation.
Wish your further comments!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3125: [FLINK-5499][JobManager]Reuse the resource location of pr...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/3125
I have actually merged this with slight adjustments to take both state location and prior inputs into account. Since batch jobs are so far stateless, this will preserve input locality for batch jobs and for the first time a streaming job is scheduled. For state resuming jobs, it will try to reuse the prior location.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3125: [FLINK-5499][JobManager]Reuse the resource location of pr...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/3125
Thank you for opening this pull request. I think the feature is a good idea, but I would like to approach it a bit broader:
- On state restore, this should prefer the old state location, agreed
- If no such location exists, it should still try to co-locate by input. Especially for the batch execution, that is quite important.
Also, this would need some tests.
I'll add some more detailed comments to the issue soon...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---