You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "MartijnVisser (via GitHub)" <gi...@apache.org> on 2026/07/03 21:15:50 UTC
[PR] [FLINK-40067][tests] Fix race in RescaleTimelineITCase.testRescaleTerminatedByJobFinished [flink]
MartijnVisser opened a new pull request, #28633:
URL: https://github.com/apache/flink/pull/28633
## What is the purpose of the change
Fixes a test-side timing race that makes `RescaleTimelineITCase.testRescaleTerminatedByJobFinished` flaky on loaded CI machines (Azure build 76571, leg `test_cron_hadoop313_core`). With the short 100ms cooldown shared by the parameterized fixture, the cooldown-driven `Idling` transition of `DefaultStateTransitionManager` terminates the in-progress rescale with `NO_RESOURCES_OR_PARALLELISMS_CHANGE` before the job finishes; `goToFinished`'s later `JOB_FINISHED` stamp is then dropped (`DefaultRescaleTimeline#updateRescale` is a no-op once the rescale is terminated), so the awaited condition can never be met. The timeout only became observable after FLINK-40009 made the wait helper's budget real. Both terminal reasons are legitimate product behaviour; this is not a product bug.
## Brief change log
- Skip the disabled-history parameter up front so it does not pay for a cluster rebuild.
- Rebuild the fixture cluster with a 60s executing cooldown/stabilization via the existing `rebuildClusterWithExecutingTimeouts` helper, keeping the in-progress rescale alive past the unblock-to-finish window (mirrors the sibling fixes FLINK-39903/FLINK-40010).
- Widen the final wait budget for headroom on loaded CI legs.
## Verifying this change
This change is already covered by existing tests. Ran `testRescaleTerminatedByJobFinished` 4x and the full `RescaleTimelineITCase` (30 run, 0 failures, 8 skipped); the method now completes in under 2 seconds, confirming `JOB_FINISHED` is stamped while the rescale is still in progress rather than waiting out any timer. The original failure needs a starved JobManager thread on a loaded CI machine and is not reproducible locally; the fix removes the dependency on that timer ordering entirely.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude Opus 4.8, via Claude Code)
Generated-by: Claude Opus 4.8 (1M context)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40067][tests] Fix race in RescaleTimelineITCase.testRescaleTerminatedByJobFinished [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on code in PR #28633:
URL: https://github.com/apache/flink/pull/28633#discussion_r3581377555
##########
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/adaptive/timeline/RescaleTimelineITCase.java:
##########
@@ -269,15 +269,24 @@ void testRescaleTerminatedBySucceeded() {}
@TestTemplate
void testRescaleTerminatedByJobFinished() throws Exception {
+ // This case only asserts on the recorded rescale history; skip the disabled-history
+ // parameter before the cluster rebuild below so it does not pay for an unused cluster.
+ assumeThat(enabledRescaleHistory(configuration)).isTrue();
Review Comment:
Thanks, I'll take a look at that!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40067][tests] Fix race in RescaleTimelineITCase.testRescaleTerminatedByJobFinished [flink]
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #28633:
URL: https://github.com/apache/flink/pull/28633#discussion_r3581194039
##########
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/adaptive/timeline/RescaleTimelineITCase.java:
##########
@@ -269,15 +269,24 @@ void testRescaleTerminatedBySucceeded() {}
@TestTemplate
void testRescaleTerminatedByJobFinished() throws Exception {
+ // This case only asserts on the recorded rescale history; skip the disabled-history
+ // parameter before the cluster rebuild below so it does not pay for an unused cluster.
+ assumeThat(enabledRescaleHistory(configuration)).isTrue();
Review Comment:
I created FLINK-40150 to cover the topic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40067][tests] Fix race in RescaleTimelineITCase.testRescaleTerminatedByJobFinished [flink]
Posted by "XComp (via GitHub)" <gi...@apache.org>.
XComp commented on code in PR #28633:
URL: https://github.com/apache/flink/pull/28633#discussion_r3580280388
##########
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/adaptive/timeline/RescaleTimelineITCase.java:
##########
@@ -269,15 +269,24 @@ void testRescaleTerminatedBySucceeded() {}
@TestTemplate
void testRescaleTerminatedByJobFinished() throws Exception {
+ // This case only asserts on the recorded rescale history; skip the disabled-history
+ // parameter before the cluster rebuild below so it does not pay for an unused cluster.
+ assumeThat(enabledRescaleHistory(configuration)).isTrue();
Review Comment:
It's kind of odd that we use `@TestTemplate` when we only want to run one option. Instead of using assume here, we could just run it as a @Test with the rescale history enabled.
Especially since we're rebuilding the Flink cluster within the test anyway with custom configuration.
But it is not that straight-forward. So it might be a follow-up ticket. We would improve the test runtime because we skip unnecessary MiniCluster startups not only in this test but also the other tests that use assume similarly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40067][tests] Fix race in RescaleTimelineITCase.testRescaleTerminatedByJobFinished [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28633:
URL: https://github.com/apache/flink/pull/28633#issuecomment-4879401292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "74b6b872a36e939443cf7b970e5d6d3001ae54d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "74b6b872a36e939443cf7b970e5d6d3001ae54d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 74b6b872a36e939443cf7b970e5d6d3001ae54d2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-40067][tests] Fix race in RescaleTimelineITCase.testRescaleTerminatedByJobFinished [flink]
Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser merged PR #28633:
URL: https://github.com/apache/flink/pull/28633
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org