[jira] [Created] (HDDS-17008) Duplicate container resolved by another volume's ContainerReader at datanode start can leave a reservation for the deleted copy

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17008:
---------------------------------

             Summary: Duplicate container resolved by another volume's ContainerReader at datanode start can leave a reservation for the deleted copy
                 Key: HDDS-17008
                 URL: https://issues.apache.org/jira/browse/HDDS-17008
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: CR-15-reserve-before-container-visible-at-load.patch, TestBugCR15DuplicateSwapReservationRace.java

h3. Mechanism
At start, {{OzoneContainer.buildContainerSet}} runs one {{ContainerReader}} per volume on a shared {{ContainerSet}}. {{ContainerReader.verifyAndFixupContainerData}} adds a container to the set and only then reserves its space ({{commitSpace}}). Nothing orders these two steps against the other readers. If the same OPEN container is on another volume with a higher BCSID (or CLOSED), that volume's reader can find the first copy in the set, swap it out, delete it from disk and release its reservation in between. That release does nothing, because nothing is reserved yet. The first reader then reserves space for the copy that was just deleted.

h3. Trigger
The same container on two volumes at datanode start, the stale copy OPEN, and the stale copy's reader descheduled between {{addContainer}} and {{commitSpace}} for as long as the other reader needs for the whole duplicate handling, including removing the stale copy from disk. The precondition is rare: the disk balancer refuses to move OPEN containers, so it does not create such a duplicate. One possible route (argued, not run) is a volume that fails with an OPEN copy, the container being replicated as CLOSED to another volume of the same datanode, and the failed volume coming back at the next start. The window is narrow as well: a run of 400 duplicate pairs without timing help did not hit it.

h3. Impact
The volume of the deleted copy keeps one container size (minus the copy's used bytes) committed until the next restart, refuses that much new data, and reports it in the storage report. No data is lost.

h3. Reproduction
[^TestBugCR15DuplicateSwapReservationRace.java] ({{hadoop-hdds/container-service}}) uses two real volumes, the same container with BCSID 5 and 10, and real {{ContainerReader}}s. The only help is timing: a {{ContainerSet}} spy runs the real {{addContainer}} and then holds the stale copy's reader on a latch until the other reader has finished. Output: {{kept copy on fresh volume=true, stale copy dir exists=false, stale volume committed=5368709120, fresh volume committed=5368709120}}. A passing test means the defect is present.

h3. Patch
[^CR-15-reserve-before-container-visible-at-load.patch], against cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at 2536fe9cbc3. Released 2.1.0 and later have the same code.

The reader reserves before it adds the container to the set, so a copy is reserved from the moment another reader can see it, and a {{finally}} releases the copy's own reservation when it is not kept (duplicate not swapped, or the add or duplicate resolution failed before the copy entered the set). A failure after the copy entered the set (removing the swapped out copy, or the container ID table update) leaves the reservations as they are today: the copy in the set unreserved, the swapped out copy still reserved. When the new copy wins, the existing copy is released as before. Covered by the new {{TestContainerReader.testDuplicateSwappedWhileFirstCopyIsLoading}}, which fails without the change ({{expected: 0L but was: 5368698880L}}), and by a new check in the existing {{testMultipleContainerReader}} that each volume's committed bytes match the OPEN containers kept on it. With the change the reader, container, persistence and handler suites pass (332 tests) and checkstyle is clean.

Not covered: {{ContainerSet.addContainer}} also adds the container ID to the volume's list after the map insert, and the swap removes it from there. A slightly shifted interleaving could leave the ID of the deleted copy in the stale volume's list (argued, not run).

Found by code review of the datanode volume and per-disk RocksDB lifecycle paths, as part of the TLA+ verification effort under HDDS-15926, on commit cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org