[jira] [Assigned] (AMBARI-26640) Prevent auto-recovery from blocking cluster start operations

["Jialiang Cai (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AMBARI-26640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jialiang Cai reassigned AMBARI-26640:
-------------------------------------

    Assignee: Jialiang Cai

> Prevent auto-recovery from blocking cluster start operations
> ------------------------------------------------------------
>
>                 Key: AMBARI-26640
>                 URL: https://issues.apache.org/jira/browse/AMBARI-26640
>             Project: Ambari
>          Issue Type: Bug
>            Reporter: Jialiang Cai
>            Assignee: Jialiang Cai
>            Priority: Major
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> h2. Problem
> Ambari Agent auto-recovery commands use the same {{ActionQueue}} as server-issued execution commands, but recovery work is generated locally and is not represented in the server request graph or shown in the UI. A queued or running recovery command can therefore occupy the Agent while an operator starts the cluster or uses {{Start All Services}}. The page appears stuck even though the server operation is waiting behind hidden Agent work.
> Auto-recovery currently decides whether to start or restart a component from only that component local desired and current state. It does not honor the cluster {{RoleCommandOrder}} used by normal cluster start requests. After a server restart, Agent reconnect, heartbeat loss, or concurrent component transition, cached state can also be stale or delivered out of order. This can start downstream components before their dependencies are ready, or repeatedly schedule recovery while an operator is trying to control the cluster.
> h2. Reproduction Scenario
> # Enable auto-recovery for one or more components.
> # Stop or fail a component so the Agent queues or starts a local recovery command.
> # From Ambari, run {{Start All Services}} or another server-issued command on the same host.
> # Observe that the visible request waits while the local recovery task is not visible in the request graph.
> # Restart Ambari Server or reconnect an Agent while component states are changing. The Agent can make recovery decisions from a snapshot that does not identify its server lifecycle, version, completeness, or reporting Agent session.
> h2. Root Causes
> * Server-issued execution commands do not preempt queued or active auto-recovery commands. There is also a dequeue-to-active race where recovery can start immediately before a server command reaches the queue.
> * Recovery configuration contains enabled components and desired state, but no dependency graph or runtime dependency state.
> * The Agent cannot distinguish a complete current-session component snapshot from cached status left by an earlier connection.
> * Recovery topology updates have no server epoch or monotonic version, so delayed updates can overwrite newer state.
> * Status acknowledgements and cached reports can cross a reconnect boundary unless they are associated with a snapshot generation.
> h2. Proposed Behavior
> * A server-issued {{EXECUTION_COMMAND}} must take priority over Agent auto-recovery. Queued recovery commands should be removed and active recovery commands should be cancelled before the server command executes. Recovery must also yield when a server command wins the dequeue-to-active race.
> * Ambari Server should derive recovery dependencies from the same transitive {{RoleCommandOrder}} used by the stage planner. Each dependency must be expanded to real component instances and include service, component, host, current state, desired state, required state, and freshness.
> * The topology sent to Agents must include a server epoch, a monotonic version, and a completeness flag. Cached server topology must not regress to an older version in the same epoch.
> * Component state freshness must be scoped to the active Agent session. Registration invalidates the previous snapshot, a complete component-status scan makes the new session fresh, heartbeat loss invalidates it again, and reports from superseded sessions are ignored.
> * Agent {{START}} and {{RESTART}} recovery must remain blocked until the topology is complete and every dependency is fresh and satisfies its required state. A {{STARTED}} component also satisfies an {{INSTALLED}} dependency. Non-start recovery behavior should remain compatible with the existing recovery mode.
> * Agent reconnect must clear the old reported-status cache before exposing the new registered connection. Acknowledgements from an earlier snapshot generation must not complete or repopulate the new snapshot.
> h2. Acceptance Criteria
> # {{Start All Services}} and other server-issued operations are not blocked behind queued or running auto-recovery work on an Agent.
> # Auto-recovery {{START}} and {{RESTART}} follow transitive cluster startup ordering across hosts and services.
> # Recovery does not run from incomplete, stale, superseded-session, or out-of-order topology data.
> # Dependency state changes are republished and eventually unblock downstream recovery after upstream components become ready.
> # Server restart, Agent reconnect, and heartbeat loss conservatively block recovery until a fresh complete status snapshot is available.
> # Existing Agents ignore the additive topology fields, and updated Agents retain compatibility with recovery configuration from an older server.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ambari.apache.org
For additional commands, e-mail: issues-help@ambari.apache.org