[jira] [Commented] (FLINK-40076) RescaleTimelineITCase.testRecordNonTerminatedRescaleMergingWithNewRecoverableFailureTriggerCause fails intermittently: snapshot taken before recoverable failover is merged into in-progress rescale

["Matthias Pohl (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-40076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094645#comment-18094645 ] 

Matthias Pohl commented on FLINK-40076:
---------------------------------------

The test was added in FLINK-38343 to Flink 2.3. Therefore, we should do a backport to 2.3

> RescaleTimelineITCase.testRecordNonTerminatedRescaleMergingWithNewRecoverableFailureTriggerCause fails intermittently: snapshot taken before recoverable failover is merged into in-progress rescale
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-40076
>                 URL: https://issues.apache.org/jira/browse/FLINK-40076
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / Coordination
>    Affects Versions: 2.3.0, 2.4.0
>            Reporter: Martijn Visser
>            Priority: Major
>              Labels: pull-request-available
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=76751&view=results (leg: test_cron_adaptive_scheduler core)
> {code}
> org.opentest4j.AssertionFailedError:
> expected: RECOVERABLE_FAILOVER
>  but was: UPDATE_REQUIREMENT
>       at org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.lambda$testRecordNonTerminatedRescaleMergingWithNewRecoverableFailureTriggerCause$17(RescaleTimelineITCase.java:582)
>       at org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.runAdaptedParameterizedAssertion(RescaleTimelineITCase.java:630)
>       at org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.testRecordNonTerminatedRescaleMergingWithNewRecoverableFailureTriggerCause(RescaleTimelineITCase.java:574)
> {code}
> The test fails fast (~0.4s), commit 3196a998cf0. Distinct from the earlier races fixed in this class (FLINK-39902, FLINK-39903, FLINK-40009, FLINK-40010) and from the open FLINK-40067, which all target other methods.
> Root cause: after {{miniCluster.terminateTaskManager(0)}} the test synchronizes via {{waitForVertexParallelismReachedAndJobRunning(..., PARALLELISM)}}. {{CommonTestUtils.waitUntilCondition}} evaluates its predicate before sleeping, and immediately after the TaskManager loss the job is still observed RUNNING at PARALLELISM, so the wait returns on stale pre-failover state and provides no synchronization (after the failover the job runs at lower parallelism, so the condition can only ever match the stale state). The {{ExecutionGraphInfo}} snapshot can therefore be taken before {{AdaptiveScheduler.recordRescaleForJobRestarting}} merges the failover into the in-progress UPDATE_REQUIREMENT rescale and re-stamps its trigger cause to RECOVERABLE_FAILOVER. Product behavior is correct; the test synchronization is wrong.
> Proposed fix: before snapshotting, wait until the failover has actually been merged (rescale history size 2 with latest trigger cause RECOVERABLE_FAILOVER), gated on the enabled-history parameterization (the disabled-history leg asserts an empty history and needs no wait). Assertions unchanged. Verified locally with 25 iterations (50 executions across both parameterizations), all green.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)