You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "yuanfenghu (Jira)" <ji...@apache.org> on 2024/04/09 02:23:00 UTC
[jira] [Comment Edited] (FLINK-35035) Reduce job pause time when cluster resources are expanded in adaptive mode
[ https://issues.apache.org/jira/browse/FLINK-35035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835106#comment-17835106 ]
yuanfenghu edited comment on FLINK-35035 at 4/9/24 2:22 AM:
------------------------------------------------------------
[~bgeng777]
Thank you for your comment. As you understand, I hope that if a new tm occurs during the running of the task, the task should not restart immediately, but wait for a period of time.
Below I will explain this process in detail:
Taking the example I gave at the beginning, assume that the current task is [v1 (maxp=10 minp = 1)] > [v2 (maxp=10, minp=1)], but the total number of slots in the cluster is 5, so the task is running [v1 p5]>[v2 p5] runs. I have now added 5 slots. I hope the task will run with [v1 p10]->[v2 p10]. When the number of cluster slots becomes 6, the task will immediately trigger a restart. At this time, according to the jobmanager.adaptive-scheduler.resource-stabilization-timeout parameter, the task will wait for a period of time during the restart phase for resources. If the slot does not reach the target slot number of 10 during this period, the task will run with a lower degree of parallelism. , but my slots will be added to 10 over a period of time, so this will trigger another expansion and restart process. In this process, I have one more restart process and one more resource waiting process. Why don't we start before the first restart? Should I wait for a period of time or determine that the number of slots meets my p=10 before triggering the restart (scale up) action?
[~echauchot] hi, can you help me look into this issue, it seems similar to FLINK-21883.
Thanks
was (Author: JIRAUSER296932):
[~bgeng777]
Thank you for your comment. As you understand, I hope that if a new tm occurs during the running of the task, the task should not restart immediately, but wait for a period of time.
Below I will explain this process in detail:
Taking the example I gave at the beginning, assume that the current task is [v1 (maxp=10 minp = 1)] -> [v2 (maxp=10, minp=1)], but the total number of slots in the cluster is 5, so the task is running [v1 p5]->[v2 p5] runs. I have now added 5 slots. I hope the task will run with [v1 p10]->[v2 p10]. When the number of cluster slots becomes 6, the task will immediately trigger a restart. At this time, according to the jobmanager.adaptive-scheduler.resource-stabilization-timeout parameter, the task will wait for a period of time during the restart phase for resources. If the slot does not reach the target slot number of 10 during this period, the task will run with a lower degree of parallelism. , but my slots will be added to 10 over a period of time, so this will trigger another expansion and restart process. In this process, I have one more restart process and one more resource waiting process. Why don't we start before the first restart? Should I wait for a period of time or determine that the number of slots meets my p=10 before triggering the restart (scale up) action?
[~echauchot] hi, can you help me look into this issue, it seems similar to FLINK-21883.
Thanks
> Reduce job pause time when cluster resources are expanded in adaptive mode
> --------------------------------------------------------------------------
>
> Key: FLINK-35035
> URL: https://issues.apache.org/jira/browse/FLINK-35035
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Affects Versions: 1.19.0
> Reporter: yuanfenghu
> Priority: Minor
>
> When 'jobmanager.scheduler = adaptive' , job graph changes triggered by cluster expansion will cause long-term task stagnation. We should reduce this impact.
> As an example:
> I have jobgraph for : [v1 (maxp=10 minp = 1)] -> [v2 (maxp=10, minp=1)]
> When my cluster has 5 slots, the job will be executed as [v1 p5]->[v2 p5]
> When I add slots the task will trigger jobgraph changes,by
> org.apache.flink.runtime.scheduler.adaptive.ResourceListener#onNewResourcesAvailable,
> However, the five new slots I added were not discovered at the same time (for convenience, I assume that a taskmanager has one slot), because no matter what environment we add, we cannot guarantee that the new slots will be added at once, so this will cause onNewResourcesAvailable triggers repeatedly
> ,If each new slot action has a certain interval, then the jobgraph will continue to change during this period. What I hope is that there will be a stable time to configure the cluster resources and then go to it after the number of cluster slots has been stable for a certain period of time. Trigger jobgraph changes to avoid this situation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)