You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "XComp (via GitHub)" <gi...@apache.org> on 2024/06/07 11:16:01 UTC
[PR] [FLINK-35550][runtime] Move rescaling functionality into dedicated class RescaleManager [flink]
XComp opened a new pull request, #24909:
URL: https://github.com/apache/flink/pull/24909
## What is the purpose of the change
The purpose of this PR is the reorganization of responsibilities for rescaling.
## Brief change log
[Class diagrams](https://app.diagrams.net/#G13IwZnJrZ_NBxkDYdAUz2IydP3A5qv8uH#%7B%22pageId%22%3A%22C5RBs43oDa-KdzZeNtuy%22%7D)
* Introduction of new interfaces `RescaleManager`, `RescaleManager.Context` and `RescaleManager.Factory`
* Responsibilities:
* `AdaptiveScheduler` only provides the available parallelism (through the `SlotManager`). The rescalingControllers moved into `RescaleManager`
* `Executing` is only in charge of state transitition and savepoints
* `RescaleManager` handles the rescale decisions
## Verifying this change
The tests were reorganized accordingly. Some additional unit tests are added to improve coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35550][runtime] Move rescaling functionality into dedicated class RescaleManager [flink]
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on PR #24909:
URL: https://github.com/apache/flink/pull/24909#issuecomment-2186460177
args, I forgot the commit on move the factory instantiation into the `AdaptiveScheduler` constructor (https://github.com/apache/flink/pull/24909/commits/1fca0a10ff19d038dbac3722cf5047de4fba45fc). The subsequent force-push squashes the changes once more ([diff](https://github.com/apache/flink/compare/1fca0a10ff19d038dbac3722cf5047de4fba45fc..1993e33efdd50f42547ffd36385835110a4d1169)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35550][runtime] Move rescaling functionality into dedicated class RescaleManager [flink]
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on PR #24909:
URL: https://github.com/apache/flink/pull/24909#issuecomment-2186462260
Final force-push to rebase to most-recent `master`. Can you approve this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35550][runtime] Move rescaling functionality into dedicated class RescaleManager [flink]
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on PR #24909:
URL: https://github.com/apache/flink/pull/24909#issuecomment-2186454160
Thanks for your reviews, @ztison and @1996fanrui . I addressed your changes and squashed the commits in a separate force-push (see [diff](https://github.com/apache/flink/compare/e365b8860b2f0bb95865d0d9fd67e2d0fbf60e1d..d5365715f4f81b2d6aabdcafdbb737c8e634d787)).
@1996fanrui Yes, this PR is solely about collecting the rescale-related code in the `Executing` state to prepare for the follow-up PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35550][runtime] Move rescaling functionality into dedicated class RescaleManager [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #24909:
URL: https://github.com/apache/flink/pull/24909#issuecomment-2154631729
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3f6cad0f5bc79ea12d5e83226c94c7e17d603339",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3f6cad0f5bc79ea12d5e83226c94c7e17d603339",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3f6cad0f5bc79ea12d5e83226c94c7e17d603339 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35550][runtime] Move rescaling functionality into dedicated class RescaleManager [flink]
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #24909:
URL: https://github.com/apache/flink/pull/24909#discussion_r1650914361
##########
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/AdaptiveScheduler.java:
##########
@@ -1048,8 +1044,8 @@ public void goToExecuting(
this,
userCodeClassLoader,
failureCollection,
- settings.getScalingIntervalMin(),
- settings.getScalingIntervalMax()));
+ DefaultRescaleManager.Factory.fromSettings(settings),
Review Comment:
sure, good point. That's possible. 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org