[jira] [Resolved] (FLINK-40969) Rescale details endpoint returns 404 for latest events evicted from AdaptiveScheduler history

["Martijn Visser (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-40969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn Visser resolved FLINK-40969.
------------------------------------
    Resolution: Fixed

> Rescale details endpoint returns 404 for latest events evicted from AdaptiveScheduler history
> ---------------------------------------------------------------------------------------------
>
>                 Key: FLINK-40969
>                 URL: https://issues.apache.org/jira/browse/FLINK-40969
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / REST, Runtime / Web Frontend
>    Affects Versions: 2.3.0
>         Environment: Affected code paths were confirmed in:
> * Apache Flink 2.3.0
> Configuration:
> * Streaming job
> * AdaptiveScheduler enabled
> * web.adaptive-scheduler.rescale-history.size set to a positive bounded value
> The issue is deployment-mode independent because it is caused by the in-memory rescale snapshot and REST lookup logic.
>            Reporter: Hao Yu
>            Assignee: Hao Yu
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 2.3.1, 2.4.0
>
>         Attachments: PixPin_2026-10-12_17-39-13.png
>
>
> h2. Problem
> The AdaptiveScheduler Rescales Overview may expose a rescale UUID whose details can no longer be retrieved.
> The Overview keeps the latest rescale event for each terminal state (COMPLETED, FAILED, and IGNORED). These entries are stored separately from the bounded rescale history.
> When a latest event for one terminal state is older than the configured history limit, newer events of other terminal states can evict it from the history. The event remains visible in the Overview, but requesting its details returns HTTP 404:
> {code:java}
> Could not find rescale details of specified rescaleUuid <rescale-uuid>.
> {code}
> The Web UI automatically requests details for the latest events returned by the Overview. Consequently, opening the Rescales page can display a server error even though the job is running normally and the UUID was returned by the Overview endpoint.
> h2. Steps to reproduce
>  # Configure a streaming job to use the AdaptiveScheduler and enable a small rescale history:
> {code:java}
> jobmanager.scheduler: adaptive
> web.adaptive-scheduler.rescale-history.size: 1
> {code}
>  # Trigger a rescale that terminates as IGNORED, for example by updating the resource requirements again while a previous rescale is still in progress.
>  # Trigger a later rescale that terminates as COMPLETED.
>  # Request:
> {code:java}
> GET /jobs/:jobid/rescales/overview
> {code}
>  # Record the UUID returned in {{{}latest.ignored{}}}.
>  # Request:
> {code:java}
> GET /jobs/:jobid/rescales/history
> {code}
> The ignored event is no longer present because the bounded history only retains the later completed event.
>  # Request:
> {code:java}
> GET /jobs/:jobid/rescales/details/:ignored-rescale-uuid
> {code}
> h2. Actual result
> The details endpoint returns HTTP 404:
> {code:java}
> Could not find rescale details of specified rescaleUuid <ignored-rescale-uuid>.
> {code}
> The Rescales Web UI surfaces this response as a server error.
> h2. Expected result
> Every rescale event returned by {{/jobs/:jobid/rescales/overview}} should remain queryable through the details endpoint while it is exposed by the Overview.
> The configured history size should continue to limit the number of entries returned by the History endpoint.
> h2. Root cause
> {{DefaultRescaleTimeline}} stores rescale information in two independent structures:
>  * {{rescaleHistory}} is a {{BoundedFIFOQueue}} limited by {{{}web.adaptive-scheduler.rescale-history.size{}}}.
>  * {{latestRescales}} retains the latest event for each terminal state independently of that limit.
> {{RescalesStatsSnapshot}} copies both structures, but {{idToRescaleMap}} is built only from {{{}rescaleHistory{}}}:
> {code:java}
> this.idToRescaleMap =
>         rescaleHistory.stream()
>                 .collect(
>                         Collectors.toMap(
>                                 r -> r.getRescaleIdInfo().getRescaleUuid(),
>                                 Function.identity()));
> {code}
> {{JobRescaleDetailsHandler}} uses this map to resolve the UUID. Therefore, an event can still be returned by the Overview through {{latestRescales}} while being absent from the details lookup.
> The archived details have the same inconsistency because {{JobRescaleDetailsHandler#archiveJsonWithPath}} only archives entries from {{{}rescaleHistory{}}}.
> h2. fix
>  * Make {{RescalesStatsSnapshot#getRescale}} fall back to {{latestRescales}} when the UUID is not found in the history index.
>  * Archive the union of {{rescaleHistory}} and {{{}latestRescales{}}}, deduplicated by rescale UUID.
>  * Optionally make the Web UI handle a transient details 404 gracefully. This should be an additional safeguard rather than the primary fix.
> This preserves the configured History limit while ensuring that entries exposed by the Overview remain queryable.
> h2. Tests
> Add regression tests covering the following cases:
>  * An IGNORED event is evicted from the bounded history but remains {{{}latest.ignored{}}}; its details can still be retrieved.
>  * Archived JSON contains details for the evicted latest event.
>  * An event present in both History and latest is archived only once.
>  * An unknown UUID still returns HTTP 404.
>  * The History endpoint continues to respect the configured size limit.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)