You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "MartijnVisser (via GitHub)" <gi...@apache.org> on 2026/06/12 13:38:07 UTC
[PR] [FLINK-38534][runtime/tests] Fix LocalRecoveryTest racing the async task deployment [flink]
MartijnVisser opened a new pull request, #28407:
URL: https://github.com/apache/flink/pull/28407
## What is the purpose of the change
`LocalRecoveryTest.testStateSizeIsConsideredForLocalRecoveryOnRestart` failed in [build 75865](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=75865) (leg `test_cron_azure core`) with `FlinkException: Exhausted retry attempts` from `SchedulerTestingUtils.waitForCheckpointInProgress` after ~65s.
Root cause: the test forces all executions to RUNNING via `setAllExecutionsToRunning` while the AdaptiveScheduler is still deploying them. Deployment builds the task deployment descriptor on an I/O thread and applies it back on the main thread, which rejects deployment once the execution has left DEPLOYING. When the forced RUNNING transition wins that race, the vertex fails with "execution state has switched to RUNNING during task restore offload", the job restarts, and the manually triggered checkpoint never registers.
This reuses FLINK-38534: the earlier fix there added `waitForAllTasksRunning` *after* the forced transition, which only observes the forced state and does not prevent the collision with the in-flight descriptor creation. This is a test-only race; the production `state != DEPLOYING` check is correct.
## Brief change log
- Add `SchedulerTestingUtils#waitForAllTasksDeploymentDescriptorsCreated(ExecutionGraph)` and call it in `LocalRecoveryTest` before forcing executions to RUNNING.
- The descriptor future is assigned only partway through `Execution.deploy()`, after the execution has already moved to DEPLOYING, so a poller on another thread can observe DEPLOYING before the future exists. The wait therefore keys off the future itself: it polls `getTddCreationDuringDeployFuture()`, treating the `IllegalStateException` thrown while the future is unset as "not ready", and joins once available. This keeps the helper race-free regardless of which thread calls it (the Javadoc documents when to prefer it over the existing `ExecutionUtils#waitForTaskDeploymentDescriptorsCreation`, which assumes the deploy step has already completed).
## Verifying this change
This change is already covered by existing tests: `LocalRecoveryTest`, run 5 times consecutively (5/5 pass).
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no (test-only change)
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude Opus 4.8 via Claude Code)
Generated-by: Claude Opus 4.8 (1M context)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38534][runtime/tests] Fix LocalRecoveryTest racing the async task deployment [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #28407:
URL: https://github.com/apache/flink/pull/28407#issuecomment-4732907343
> Hi, @MartijnVisser Could you help rebase the latest master branch ,then re-trigger the CI?
@RocMarshal Done, can you give it another pass?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38534][runtime/tests] Fix LocalRecoveryTest racing the async task deployment [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser merged PR #28407:
URL: https://github.com/apache/flink/pull/28407
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38534][runtime/tests] Fix LocalRecoveryTest racing the async task deployment [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #28407:
URL: https://github.com/apache/flink/pull/28407#issuecomment-4699309538
Hi, @MartijnVisser Could you help rebase the latest master branch ,then re-trigger the CI?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-38534][runtime/tests] Fix LocalRecoveryTest racing the async task deployment [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28407:
URL: https://github.com/apache/flink/pull/28407#issuecomment-4691853614
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "4e618d8c8d711ac1830dbe7c935497f46d6e3871",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4e618d8c8d711ac1830dbe7c935497f46d6e3871",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4e618d8c8d711ac1830dbe7c935497f46d6e3871 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org