You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Izeren (via GitHub)" <gi...@apache.org> on 2025/07/14 15:24:25 UTC
[PR] [FLINK-37701][flink-runtime] Backporting of the fix to 2.0 for AdaptiveScheduler ignoring checkpoint states sizes for local recovery adjustment. [flink]
Izeren opened a new pull request, #26796:
URL: https://github.com/apache/flink/pull/26796
Previously merged PRs:
* https://github.com/apache/flink/pull/26663
* https://github.com/apache/flink/pull/26712
# PR1 https://github.com/apache/flink/pull/26663
## What is the purpose of the change
Address local recovery issues when Adaptive scheduler is enabled.
1. Pass latest completed checkpoint in addition to execution graph to `StateSizeEstimates` (that is needed because execution graph goes through cancelling/cancelled state and checkpoint coordinator is `nulled` by the time we run calculations).
2. Assign positive priority score to allocations that have overlapping key groups even when state size is zero (currently we would only give priority score if `managedKeyedState` is present, but local recovery semantics doesn't `require` state presence).
**Context**: When job can be recovered locally, we should keep slot allocation after restart to maintain
## Verifying this change
`LocalRecoveryITCase#testRecoverLocallyFromProcessCrashWithWorkingDirectory` now passes when `AdaptiveScheduler` is enabled.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (**yes** / no / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
# PR2 https://github.com/apache/flink/pull/26712
## Brief change log
* Extract commonly re-used elements from `AdaptiveSchedulerTest` into `AdaptiveSchedulerTestBase`.
* Move local recovery test into independent class inheriting from `AdaptiveSchedulerTestBase`.
* Restore checkstyle file length limit to 3100 lines.
* minor cleanup changes missed in previous PR
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37701][flink-runtime] Backporting of the fix to 2.0 for AdaptiveScheduler ignoring checkpoint states sizes for local recovery adjustment. [flink]
Posted by "Izeren (via GitHub)" <gi...@apache.org>.
Izeren commented on PR #26796:
URL: https://github.com/apache/flink/pull/26796#issuecomment-3072691741
Thank you Rui
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37701][flink-runtime] Backporting of the fix to 2.0 for AdaptiveScheduler ignoring checkpoint states sizes for local recovery adjustment. [flink]
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui merged PR #26796:
URL: https://github.com/apache/flink/pull/26796
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37701][flink-runtime] Backporting of the fix to 2.0 for AdaptiveScheduler ignoring checkpoint states sizes for local recovery adjustment. [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26796:
URL: https://github.com/apache/flink/pull/26796#issuecomment-3070029113
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "25bab16f303f8f3fcce2324a771d22d0a8954c1a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25bab16f303f8f3fcce2324a771d22d0a8954c1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25bab16f303f8f3fcce2324a771d22d0a8954c1a UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org