[jira] [Assigned] (FLINK-40379) AdaptiveScheduler is in restart loop when task cancellation consistently times out

["Anton Kalashnikov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-40379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Anton Kalashnikov reassigned FLINK-40379:
-----------------------------------------

    Assignee: Anton Kalashnikov

>  AdaptiveScheduler is in restart loop when task cancellation consistently times out
> -----------------------------------------------------------------------------------
>
>                 Key: FLINK-40379
>                 URL: https://issues.apache.org/jira/browse/FLINK-40379
>             Project: Flink
>          Issue Type: Bug
>            Reporter: Anton Kalashnikov
>            Assignee: Anton Kalashnikov
>            Priority: Major
>
> When task cancellation keeps timing out during a scale-up, the affected TM is killed, the cluster ends up with fewer slots than before, and the AdaptiveScheduler loops forever in Restarting -> WaitingForResources -> CreatingExecutionGraph -> Restarting, never making progress
> *Root cause*
> WaitingForResources, doesn't wait for resources long enough after the TM was canceled(if TM is stuck in canceled, it will be terminated in a hard way and it needs some time to restart it). So the stabilization timeout for after restart is normally small to avoid downtime in normal cases, and "sufficient resources" means slots >= lowerBound but lowerBound is hard-coded to 1, so the gate is always open
> *result:*
> The scheduler immediately re-enters CreatingExecutionGraph -> Executing, cancellation times out again, the TM is killed, and the loop continues with the same or fewer slots
> State transition entry point: Restarting.java:116 in confluentinc/flink (onGloballyTerminalState handler)
> *Fix proposal:*
> Adding `AdaptiveScheduler#getFreeSlotsVertexParallelism()`, computed from free slots only (`DeclarativeSlotPool#getFreeSlotTracker`), excluding slots still reserved by the execution that is being canceled. WaitingForResources now uses this (via a targetParallelism gate) to redefine "desired resources" only for rescale-triggered restarts: desired means the free-slot-achievable parallelism is at least what was free right before the restart. The onFailure (TM/slot loss) and reconfigure paths are untouched: they never carry a restart target, so they keep the original all-slots-based hasSufficientResources()/hasDesiredResources() unchanged
> Adding a new dedicated option `jobmanager.adaptive-scheduler.rescale.resource-stabilization-timeout` (default 2 min), used only for rescale-triggered restarts. It bounds how long the scheduler waits, after a rescale-restart, for free slots to reach the pre-restart parallelism before falling back to running with whatever's currently sufficient. Keeping it separate from the submission path means initial job-submission latency is unaffected
> *Behavior comparison:* 
> ||Scenario||Resource situation||master||This branch (final)||Difference||
> |Initial submission|Full desired reached in time|Don't wait as soon as sufficient|Identical - this path never carries a restart target, code untouched|None|
> |Initial submission|Only sufficient (< desired) available|After sufficient resources appear, waits 10s for desired, then runs with sufficient|Identical|None|
> |Scale up (RESCALE)|Free slots catch up to pre-restart target quickly|All-slots check usually (incorrectly) reports "unchanged" right after cancel and jumps straight to {{{}CreatingExecutionGraph{}}}, potentially racing on slots not yet actually free|Free-slots check correctly detects the gap, enters {{{}WaitingForResources{}}}, but exits the stabilizing phase almost as fast once slots genuinely free up|Similar wall-clock time in the good case, but no longer racing ahead of real slot availability|
> |Scale up (RESCALE)|Sufficient but < pre-restart target for a while|No dedicated wait tied to the rescale target: either proceeds near-immediately on the (possibly wrong) all-slots view, or - if the mismatch is caught - waits only 10s against the job's generic desired resources (not the specific pre-restart target), then runs at whatever's sufficient|Waits up to 2 min specifically for free slots to reach the pre-restart target before falling back to "sufficient"|New code waits materially longer (2 min vs ~0-10s) for the actual intended target before settling for less - fewer premature under-scaled restarts, at the cost of up to ~2 min extra restart latency in the worst case|
> |Scale down (RESCALE)|Same two sub-cases - identical code path, direction-agnostic (only checks whether free slots reach what was achievable pre-restart)|Same as scale-up row|Same as scale-up row|Same as scale-up row|
> |TaskManager/slot failure (task exception restart)|Desired reached quickly / only sufficient available|Restart target is {{null}} here in both versions - untouched, uses 10s + all-slots desired/sufficient checks|Identical|None|
> |Reconfigure (JAR/config change restart)|Desired reached quickly / only sufficient available|Restart target is {{null}} here in both versions - untouched|Identical|None|



--
This message was sent by Atlassian Jira
(v8.20.10#820010)