You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/10/03 07:56:00 UTC
[jira] [Updated] (HDDS-16460) Container delete leaves a zombie recoveringContainerMap entry (masked by scrub null guard)
[ https://issues.apache.org/jira/browse/HDDS-16460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDDS-16460:
----------------------------------
Labels: pull-request-available (was: )
> Container delete leaves a zombie recoveringContainerMap entry (masked by scrub null guard)
> ------------------------------------------------------------------------------------------
>
> Key: HDDS-16460
> URL: https://issues.apache.org/jira/browse/HDDS-16460
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Huang Kuan Hao
> Priority: Minor
> Labels: pull-request-available
> Attachments: HDDS-16460.001.patch
>
>
> h3. Finding
> The container delete path leaves a zombie recoveringContainerMap entry (no removeRecoveringContainer call). If the mask were absent the stale entry would drive the pre-HDDS-7842 unguarded scrub to NPE; at HEAD the consequence is prevented by the scrub null guard plus non-reused container IDs plus the HDDS-15791 auto-create rejection. Reachable via the public force-delete API but a hygiene/defensive gap only.
> h3. Classification
> * Verdict: MASKED
> * Severity: Low
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-3
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: dn-ec-reconstruction
> Guidance: campaigns/ozone-9fbf9ee/targets/022-dn-ec-reconstruction/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8 \
> "dn-ec-reconstruction|apache/ozone|Java|Use the target-specific .prompt-extra.md"
> {code}
> Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+ specification, counterexample, and confirmation debate live in the Specula run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. EC reconstruction creates a target replica RECOVERING on a datanode; {{ContainerSet.addContainer}} records a {{recoveringContainerMap}} entry.
> 2. Reconstruction is abandoned or SCM issues a (force) delete for that replica, so {{KeyValueHandler.deleteContainer}} then {{deleteInternal}} runs {{containerSet.removeContainer(containerId)}}.
> 3. The container leaves the {{ContainerSet}} but its {{recoveringContainerMap}} entry lingers (a stale/zombie entry: container absent, map entry present).
> Masked consequence (why this is a latent defect, not a live bug at HEAD): the stale entry is later reaped by the scrub ({{getTasks}} removes it and the {{getContainer(id) == null}} guard skips {{markContainerUnhealthy}}), and container IDs are not reused, so no spurious UNHEALTHY results today. The guard that prevents the crash was itself added to fix a NullPointerException the scrub hit on exactly this stale-entry state (the scrub previously dereferenced the container unconditionally), so the correspondence between the recovering map and live containers is assumed by the code, not tolerated by design.
> h3. Root cause
> The two terminal exits from the RECOVERING state are asymmetric. The close path ({{KeyValueHandler.markContainerForClose}}) clears the recovering map via {{containerSet.removeRecoveringContainer}}, but the delete path ({{deleteInternal}}) calls only {{containerSet.removeContainer}} and never {{removeRecoveringContainer}}, leaving the entry behind.
> h3. Suggested fix
> Attached patch [^HDDS-16460.001.patch] adds {{containerSet.removeRecoveringContainer(containerId)}} in {{deleteInternal}} alongside {{removeContainer(containerId)}} (both run under the container write lock), mirroring {{markContainerForClose}}, so both terminal RECOVERING exits keep the recovering map consistent with live containers. A unit test force-deletes a RECOVERING container through the public handler path and asserts the recovering map is empty afterward.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org