You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/10/11 07:44:00 UTC

[jira] [Created] (HDDS-16897) Datanode runs a resent DeleteContainerCommand against a replica imported after the command arrived, so a balancer move can leave the container under replicated while reporting COMPLETED

Siyao Meng created HDDS-16897:
---------------------------------

             Summary: Datanode runs a resent DeleteContainerCommand against a replica imported after the command arrived, so a balancer move can leave the container under replicated while reporting COMPLETED
                 Key: HDDS-16897
                 URL: https://issues.apache.org/jira/browse/HDDS-16897
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-6-skip-delete-of-container-added-after-command-received.patch, TestBugMC6StaleDeleteWipesMoveTarget.java

h3. Mechanism
SCM completes a pending DELETE op as soon as a report says the replica is gone ({{ContainerStateManagerImpl.removeContainerReplica}} calls {{ContainerReplicaPendingOps.completeDeleteReplica}}), whichever command removed it. Since HDDS-12127, {{ReplicationManager.opCompleted}} resends an expired DELETE as the same command with a new deadline. If the first copy already ran on the datanode but its DELETED report reaches SCM only after the SCM deadline, SCM resends the command, and the late report then completes the op while the resent copy is still queued on the datanode. SCM no longer knows about that copy.

{{MoveManager.move}} checks the replicas and pending ops SCM knows of, but not commands already handed to the datanode, so it accepts that datanode as a move target. When the copy has been imported and reported, {{MoveManager.handleSuccessfulAdd}} sees a healthy replica set and deletes the source. The queued command then runs: {{DeleteContainerCommandHandler.handleInternal}} checks only the deadline and the SCM term and deletes by container ID, so it force deletes the replica the move just created.

h3. Trigger
With the default timeouts ({{hdds.scm.replication.event.timeout}} 12 minutes, datanode offset 6 minutes):
# A CLOSED RATIS/THREE container has 4 replicas. ReplicationManager sends a force DELETE to datanode V.
# V runs it, but cannot reach SCM for long enough that the DELETED report arrives after the 12 minute SCM deadline (so at least about 6 minutes, and less than the dead node interval).
# The scrubber resends the DELETE to V.
# V reconnects. The heartbeat carries the late report and the response carries the resent DELETE, which waits behind other work in V's delete container threads (2 by default).
# Within the 6 minute datanode deadline of the resent command, the balancer moves this container to V, the copy completes and MoveManager deletes the source.
# The queued DELETE runs on V and deletes the imported replica.

h3. Impact
The container is left with 2 of 3 replicas while the balancer is told the move {{COMPLETED}}. ReplicationManager re-replicates from the remaining two, so there is no data loss on its own, only a window with reduced redundancy. The trigger needs a long report delay, a backed up delete queue and the balancer choosing that container and datanode within a few minutes, so it is unlikely.

h3. Reproduction
PASS, deterministic, unmodified source. [^TestBugMC6StaleDeleteWipesMoveTarget.java] ({{hadoop-hdds/server-scm}}) runs the real ReplicationManager (over replication through its own health check and {{OverReplicatedProcessor}}), {{ContainerReplicaPendingOps}}, {{ContainerManagerImpl}}, {{IncrementalContainerReportHandler}} and {{MoveManager}}, and gives each datanode a real {{DeleteContainerCommandHandler}}, {{ContainerController}} and {{ContainerSet}}. Only the disk work of the container handler is mocked. The test controls when the delayed report is processed and keeps V's delete thread (1 thread in the test) busy until the moved replica is imported. Output on the unmodified source:
{noformat}
late ICR completed (removed) the pending DELETE, DELETE #2 still queued on V (V=d2) | scmReplicas=[d1, d3, d4] pendingOps=[] dnHas=[d1,d3,d4] V.deleteQueue=1
MoveManager accepted move d1 -> d2 (V=d2) | scmReplicas=[d1, d3, d4] pendingOps=[ADD@d2] dnHas=[d1,d3,d4] V.deleteQueue=1
ADD completed, MoveManager sent DELETE to source d1 (V=d2) | scmReplicas=[d1, d2, d3, d4] pendingOps=[DELETE@d1] dnHas=[d1,d2,d3,d4] V.deleteQueue=1
V delete thread drained, delete calls on V = 2 (V=d2) | scmReplicas=[d1, d2, d3, d4] pendingOps=[DELETE@d1] dnHas=[d1,d3,d4] V.deleteQueue=0
RESULT: move=COMPLETED dnReplicas=2 scmReplicas=[d3, d4] health=UNDER_REPLICATED (RF 3)
{noformat}
A passing test means the defect is present. The heartbeat transport and the datanode disk work are simulated.

h3. Suggested fix
The patch makes the datanode skip a delete for a container that was not present when the command arrived but is present when it runs. Other ways a queued delete can meet a newer replica (the replica disappears for another reason, such as a failed volume, while the original command is still queued) are not covered. An SCM side alternative would keep a resent DELETE's target excluded from moves until the command's datanode deadline has passed, which needs new state in SCM; it is not part of this patch.

h3. Patch
[^MC-6-skip-delete-of-container-added-after-command-received.patch], against b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at 6818707698c.

{{DeleteContainerCommandHandler.handle}} records whether the container exists when the command is received. If it did not, and it exists when the command runs, the command is skipped with an INFO log. In every other case the behaviour is unchanged: an absent container is still a no-op, and a replica that was present on receipt is deleted as before, including when the container object is replaced in between (for example by the disk balancer). No protocol change.

The resend comes from HDDS-12127. HDDS-16447 is the same race on the pending op in the opposite order (a report removes the op, then the resend inserts it again); both follow from SCM tracking a DELETE per target and replica index rather than per command. This patch also keeps the orphan resends described there from deleting a later replica on that datanode, because the container is absent when they arrive. The linked issue covers the other defect of the same resend (sent without rechecking that other copies remain), which needs an SCM side fix.

Covered by {{testContainerAddedAfterCommandReceivedIsNotDeleted}} and {{testContainerPresentWhenCommandReceivedIsDeleted}} in the existing {{TestDeleteContainerCommandHandler}}. Without the change the first fails ({{NeverWantedButInvoked}} on {{deleteContainer}}). With it the datanode state machine and command handler unit suites pass and checkstyle is clean. With the patch the reproduction fails: the queued DELETE is skipped, the moved replica survives and the container ends with 3 replicas, HEALTHY.

Found by TLA+ model checking and code review of the SCM container move and balancer paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org