[jira] [Updated] (HDDS-17061) Recon drops the replicas of a DELETED container when it moves it back to SCM's live state

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17061:
------------------------------
    Attachment: TestBugMC10RebuildDropsReplicas.java

> Recon drops the replicas of a DELETED container when it moves it back to SCM's live state
> -----------------------------------------------------------------------------------------
>
>                 Key: HDDS-17061
>                 URL: https://issues.apache.org/jira/browse/HDDS-17061
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Minor
>         Attachments: MC-10-keep-replicas-when-recon-reopens-deleted-container.patch, TestBugMC10RebuildDropsReplicas.java
>
>
> h3. Mechanism
> When Recon has a container DELETED but SCM lists it as CLOSED or QUASI_CLOSED, {{ReconStorageContainerSyncHelper.rebuildContainerFromScm}} calls {{deleteContainer}}, which removes the whole entry including its replica set, and then {{addNewContainer}} with SCM's {{ContainerInfo}}. Two symptoms follow:
> # The re-added container has no replicas. The next health sweep ({{ReconReplicationManager.processAll}}) records it MISSING although datanodes still hold it, at the latest until each datanode's next full container report ({{hdds.container.report.interval}}, default 60m) adds its replica back.
> # If {{addNewContainer}} throws, the failure is only logged and the container is no longer in Recon's map. {{storeHealthStatesToDatabase}} rewrites UNHEALTHY_CONTAINERS rows only for containers in the map, so a row the container had before stays. {{ContainerEndpoint.toUnhealthyMetadata}} then gets {{ContainerNotFoundException}} for it and {{/api/v1/containers/unhealthy}} and {{/api/v1/containers/unhealthy/MISSING}} fail with HTTP 500 for every page that includes the row, until a later sync adds the container back.
> SCM's own report handler moves a container back from DELETED in place with {{ContainerManager.transitionDeletingOrDeletedToTargetState}}, keeping the record and its replicas.
> h3. Trigger
> The rebuild runs only when Recon has the container DELETED while SCM lists it as live, and Recon has not moved it back itself. Recon's report handler moves it back in place when it gets the same datanode report that made SCM do so, so the state needs Recon to miss or misjudge that report. The route that was run: Recon's DELETED pass reads a page of SCM's DELETED IDs, a datanode reports a non-empty CLOSING replica that makes SCM move the container to QUASI_CLOSED (HDDS-14684) before the page is processed, Recon retires the container from the stale page while keeping the replica, and the datanode then closes its replica (bcsId not above the container's), which does not move a DELETED container back. The next sync cycle runs the rebuild. Another route, argued and not run: Recon is down while the datanode reports the replica that makes SCM move the container back.
> Symptom 2 also needs {{addNewContainer}} to fail and a MISSING row from before the container was retired, which survives only if no health sweep ran between the two syncs. At defaults the sweep runs every 5 minutes ({{ozone.recon.task.missingcontainer.interval}}) and the sync every 6 hours, so this needs a second sync within minutes of the one that retired the container: a sync interval configured below the sweep interval, or a Recon restart, where the first sync runs 1 minute after startup (argued, not run). For a CLOSED or QUASI_CLOSED container {{addNewContainer}} can fail only on a container table write error or if Recon's pipeline sync removes the container's pipeline between the pipeline check and the pipeline update inside {{ContainerStateManagerImpl.addContainer}}. Neither was driven; the reproduction injects the exception.
> h3. Impact
> * Symptom 1: a false MISSING container in Recon's unhealthy containers API and UI for up to one full report interval. No data is lost; the replica history in Recon's DB is kept. The same code is in 2.2.0 and 2.2.1.
> * Symptom 2: the unhealthy containers API returns HTTP 500 until the next successful sync ({{ozone.recon.scm.container.sync.task.interval.delay}}, default 6h).
> h3. Reproduction
> Both PASS, deterministic, unmodified source, in {{hadoop-ozone/recon}}. A passing test means the defect is present.
> * [^TestBugMC10RebuildDropsReplicas.java]: real {{ReconContainerManager}}, {{ReconContainerReportHandler}} fed with full reports, {{SCMNodeManager}}, sync helper and health sweep (its database writes captured by a mock); only the SCM client is a fake backed by a state table, and the SCM state change and datanode report are placed right after the DELETED page RPC returns. For SCM QUASI_CLOSED and for SCM CLOSED, the rebuild leaves 0 replicas and the sweep writes MISSING; the next full report restores the replica and the row goes away.
> * [^TestBugMC11StaleRowAfterFailedRebuild.java]: real container manager, sync helper, health sweep, UNHEALTHY_CONTAINERS table (Derby) and {{ContainerEndpoint}}; the SCM client is a mock and one {{IOException}} is injected into {{addNewContainer}}. The container leaves the map, its MISSING row survives three sweeps, and {{getUnhealthyContainers}} throws a {{WebApplicationException}} with status 500 caused by {{ContainerNotFoundException}}. The next sync re-adds the container and the endpoint returns 200. A control test with a sweep between the two syncs shows no failure.
> h3. Patch
> [^MC-10-keep-replicas-when-recon-reopens-deleted-container.patch], against 916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master at 62d982ff2d9.
> {{rebuildContainerFromScm}} still checks that SCM lists the container in that state, then moves Recon's record back in place with {{transitionDeletingOrDeletedToTargetState}}, as SCM's report handler does, instead of deleting and re-adding it. The replicas are kept, and if the update fails the container stays in Recon's map with its replicas, so the next sweep rewrites its rows and the next sync tries again. This fixes both symptoms. Recon keeps its own copy of the container metadata (sequence ID, key count, used bytes) instead of taking SCM's; the report handler keeps those up to date from replica reports.
> Covered by {{containerDeletedInReconButClosedInScmKeepsReplicas}} and {{failedCorrectionFromDeletedKeepsContainerInRecon}} in {{TestReconSCMContainerSyncIntegration}}. Without the change they fail (0 replicas; container gone from the map); with it the Recon {{scm}} package tests pass and checkstyle is clean. With the patch the reproductions fail at their defect assertions: the replica is kept, and {{addNewContainer}} is no longer called so the container stays in the map. The control test of the second reproduction passes with and without the patch; it only checks that the endpoint answers 200 when a sweep ran between the two syncs.
> Not changed: the stale DELETED page itself. Recon still retires such a container and shows it DELETED until the next sync moves it back. A tolerant {{ContainerEndpoint.toUnhealthyMetadata}} (skip and log a row whose container is not in the map) would be separate hardening.
> Found by TLA+ model checking and code review of the datanode container report and Recon container tracking paths under HDDS-15926, on commit 916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org