[jira] [Updated] (HDDS-17026) Datanode container create that fails with an unchecked exception leaks the reserved container size on the volume until restart

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17026:
------------------------------
    Attachment: TestBugMC10DbLoadReservationLeak.java

> Datanode container create that fails with an unchecked exception leaks the reserved container size on the volume until restart
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17026
>                 URL: https://issues.apache.org/jira/browse/HDDS-17026
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Minor
>         Attachments: MC-10-release-reservation-on-unchecked-create-failure.patch, TestBugMC10DbLoadReservationLeak.java
>
>
> h3. Mechanism
> {{KeyValueContainer.create}} reserves the container size ({{maxSize}}, 5 GB by default) on the volume that {{chooseVolume}} picks. {{KeyValueHandler.handleCreateContainer}} gives that reservation back only in its {{catch (StorageContainerException)}} branch. When {{create}} fails with an unchecked exception after the volume was chosen, the exception passes that branch, {{KeyValueHandler.handle}} turns it into a {{CONTAINER_INTERNAL_ERROR}} response, and the volume's committed bytes stay raised by the full container size. Nothing recomputes committed bytes while the datanode runs, so each such attempt adds another {{maxSize}} until restart. Present since HDDS-12810 (2.1.0), which moved the reservation into {{chooseVolume}}; 2.0.0 is not affected (checked in the ozone-2.0.0, 2.1.0 and 2.2.1 tags).
> h3. Trigger
> Today this is reached through the state described in HDDS-11485: a data volume whose per-volume DB did not load at restart (for example its {{DS-<id>}} directory or {{container.db}} is gone, or a dedicated DB volume was lost) stays in service with {{dbParentDir}} unset. Every container create placed on it, including the implicit create {{HddsDispatcher}} issues for the first write to a new container, throws {{NullPointerException("dbParentDir == null")}} from {{KeyValueContainerLocationUtil.getContainerDBFile}} after the reservation. Any other unchecked failure in {{create}} after the volume is chosen leaks the same way.
> h3. Impact
> * Each failed create leaves {{maxSize}} committed on the volume. In the reproduction three creates left 15 GB committed on an empty volume, and the storage report sent to SCM carries it.
> * The volume choosing policy counts committed bytes against the volume's free space, so the phantom bytes grow until they fill it; after that the policy stops picking the volume (by code reading, not run). On the known trigger every create on that volume fails anyway, so the harm is the wrong committed and free space figures, on the datanode and in what SCM sees.
> * The creates themselves fail either way; this report is only about the space they leave reserved.
> h3. Reproduction
> PASS on unmodified source. [^TestBugMC10DbLoadReservationLeak.java] ({{hadoop-hdds/container-service}}, target path and Maven command in the class comment) formats a data volume whose DB lives on a dedicated DB volume, loses the DB disk and drops it from the configuration, then rebuilds the volume set with a real {{StorageVolumeChecker}}, loads the DBs and runs the startup volume check. The volume is {{NORMAL}} with {{isDbLoaded=false}} and {{dbParentDir=null}}. Three CreateContainer requests go through {{KeyValueHandler.handle}}; each returns {{CONTAINER_INTERNAL_ERROR}} with "dbParentDir == null", no container is added, and committed bytes grow to 5368709120, 10737418240 and 16106127360. A later volume check still finds the volume healthy and its report carries 3 x 5 GB committed. A passing test means the defect is present.
> h3. Patch
> [^MC-10-release-reservation-on-unchecked-create-failure.patch], against cce1ca28944886218132f01c0e271c1fd0700078. It also applies to master at 2536fe9cbc3.
> {{handleCreateContainer}} now releases the reservation in its {{finally}} block whenever the container was not created, instead of only in the {{StorageContainerException}} branch. {{ContainerData.releaseCommitSpace}}, unchanged, already gives back the space only while the reservation is still held and then clears the flag, so a path that already released it is not released twice, and the branch for an already existing container, which reserves nothing, is unaffected. Volume state handling is not changed: with the patch such a volume still stays in service and the creates still fail, but nothing stays reserved. The root cause of that volume state is tracked in HDDS-11485; this fix is independent of it.
> Covered by {{testCreateContainerReleasesSpaceOnUncheckedFailure}} in the existing {{TestKeyValueHandler}}: a schema V3 volume with its working directory but no per-volume DB, one CreateContainer through {{KeyValueHandler.handle}}. Without the change it fails with "expected: 0L but was: 5368709120L"; with it the volume has no committed bytes. The new {{testCreateContainerKeepsSpaceOfCreatedContainer}} covers the guard: a successful create keeps {{maxSize}} committed with the reservation flag set, and a repeated create of the same container leaves both unchanged; it fails if the release is made unconditional. The existing {{testCreateContainerWithFailure}}, which makes {{ContainerSet.addContainer}} throw {{StorageContainerException}}, covers the checked path and passes unchanged. With the change the handler, container, dispatcher, persistence and {{OzoneContainer}} suites (551 tests) pass and checkstyle is clean. With the patch the reproduction fails at its committed bytes check after the first create, while the volume is still {{NORMAL}}.
> Found by TLA+ model checking and code review of the datanode volume and per-disk RocksDB lifecycle paths under HDDS-15926, on commit cce1ca28944886218132f01c0e271c1fd0700078. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. Related but not duplicates: HDDS-11485 (the volume state that reaches this today) and HDDS-17009 (releases the reservation on a volume that the retry loop in {{KeyValueContainer.create}} gives up on; the two patches are independent and apply together). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org