[jira] [Created] (HDDS-17021) Failed mark for close of a RECOVERING container drops it from stale recovering tracking, so it is never marked UNHEALTHY

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17021:
---------------------------------

             Summary: Failed mark for close of a RECOVERING container drops it from stale recovering tracking, so it is never marked UNHEALTHY
                 Key: HDDS-17021
                 URL: https://issues.apache.org/jira/browse/HDDS-17021
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: CR-17-keep-recovering-container-tracked-on-failed-close.patch, TestBugCR17FailedCloseUntracksRecoveringContainer.java

h3. Mechanism
{{KeyValueHandler.markContainerForClose}} removes a RECOVERING container from the stale recovering tracking map ({{ContainerSet.removeRecoveringContainer}}) before it calls {{KeyValueContainer.markContainerForClose}}. When writing the {{.container}} file then fails, {{KeyValueContainer.updateContainerData}} rolls the state back to RECOVERING, but the tracking entry is not restored. {{StaleRecoveringContainerScrubbingService}} only looks at that map, so it never marks the container UNHEALTHY, and {{ContainerSet.getContainerReport}} skips RECOVERING containers, so SCM never sees it either.

h3. Trigger
# EC reconstruction creates a RECOVERING target container and writes to it.
# The coordinator's {{CloseContainer}} fails on that target because the {{.container}} file cannot be written (EIO, ENOSPC, EACCES) while the volume stays healthy.
# The coordinator's cleanup ({{deleteContainerInState}}) does not remove the container.

In the usual case step 3 does not happen: {{ECReconstructionCoordinator.reconstructECContainerGroup}} deletes every target it created after any failure. From code reading (not run), it does not when the delete RPC fails (the error is only logged), when the coordinator, which is another target, dies before the delete, or when {{deleteInternal}} gives up on the container lock and returns success without deleting. No other close path reaches a RECOVERING container.

h3. Impact
The container stays RECOVERING in memory, untracked and unreported, until the datanode restarts ({{ContainerReader}} then marks it UNHEALTHY or deletes it). Until then a later reconstruction of the same container that picks this datanode as a target fails at container creation ({{CONTAINER_ALREADY_EXISTS}}), because SCM does not know the replica is there. It also holds its data on disk, and SCM cannot remove it. Without the bug the scrubber marks it UNHEALTHY {{ozone.recovering.container.timeout}} (default 20m) after its last write, after which it is reported and can be removed. No data loss. Affects all releases since 1.3.0 (HDDS-6987).

h3. Reproduction
PASS on unmodified source, deterministic. [^TestBugCR17FailedCloseUntracksRecoveringContainer.java] ({{hadoop-hdds/container-service}}) creates two RECOVERING containers through {{KeyValueHandler.handleCreateContainer}} on a real {{HddsVolume}} with a recovering timeout of 0. The first is closed through {{handleCloseContainer}} while its metadata directory is read only (run as a non root user), the second is the control; then one scrubber run is executed. Output: {{failed close: result=CONTAINER_FILES_CREATE_ERROR state=RECOVERING inRecoveringMap=false}} and {{after scrub: control state=UNHEALTHY, failed state=RECOVERING, containers in report=[2]}}. A passing test means the defect is present. Not exercised: a real EIO or ENOSPC, the reconstruction coordinator, SCM.

h3. Patch
[^CR-17-keep-recovering-container-tracked-on-failed-close.patch], against cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at 2536fe9cbc3.

The tracking entry is now removed only after the state change to CLOSING succeeded, under the same container write lock. On failure the entry stays with its original deadline, so the scrubber still marks the container UNHEALTHY in time. Restoring the entry on failure was not chosen, because the existing helper resets the deadline. The {{logRecovered}} container log line is unchanged. On a successful close the entry now stays in the map during the {{.container}} write, so a container whose deadline has already passed can be picked up by a scrubber run in that window and later marked UNHEALTHY although it closed. That is the existing HDDS-16455 race widened by one file write, and its proposed fix (check the state under the container lock in the scrub task) covers it. Covered by the new {{TestKeyValueHandler.testFailedCloseKeepsRecoveringContainerTracked}}, which fails without the change (empty tracking map after the failed close) and also checks that a successful close drops the entry. With the change the handler, scrubber, container, persistence, reader, container set and close command suites pass and checkstyle is clean. With the patch the reproduction fails, because the container is tracked and marked UNHEALTHY like the control.

Related, not duplicates: HDDS-16460 (stale entry left after delete), HDDS-16455 and HDDS-16456 (races between the scrubber, the timeout update and a successful close).

Found by code review of the datanode volume and per-disk RocksDB lifecycle paths, as part of the TLA+ verification effort under HDDS-15926, on commit cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org