[jira] [Updated] (HDDS-17045) Recon adds a Legacy directory's totals to its ancestors again on every in place update of the directory key, such as an ACL change

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17045:
------------------------------
    Attachment: TestBugCR3LegacyDirUpdateDoubleCount.java

> Recon adds a Legacy directory's totals to its ancestors again on every in place update of the directory key, such as an ACL change
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17045
>                 URL: https://issues.apache.org/jira/browse/HDDS-17045
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: CR-3-resolve-old-parent-of-legacy-directory-update.patch, TestBugCR3LegacyDirUpdateDoubleCount.java
>
>
> h3. Mechanism
> For an UPDATE event on a directory key in a LEGACY bucket with {{ozone.om.enable.filesystem.paths=true}}, {{NSSummaryTaskWithLegacy.processWithFileSystemLayout}} builds the old directory with {{oldKeyInfo.getParentObjectID()}}. keyTable entries do not carry a parent object ID (it is 0), which is why the same method looks the parent up by name with {{setKeyParentID}} for the new value and for files. {{NSSummaryTaskDbEventHandler.handleDeleteDirEvent}} then finds no summary for parent 0, logs "The namespace table is not correctly populated." and returns without subtracting anything, while {{handlePutDirEvent}} adds the directory's existing totals to its real parent and all ancestors.
> FSO and OBS buckets, Legacy buckets without filesystem paths, and Legacy file keys resolve the old parent correctly.
> h3. Trigger
> LEGACY bucket, {{ozone.om.enable.filesystem.paths=true}} (not the default). Create directory {{dir1}} and a file {{dir1/file1}}, then change the ACL of the key {{dir1/}} with {{ObjectStore.addAcl}}. The other requests that rewrite a directory key in place ({{setAcl}}, {{removeAcl}}, {{setTimes}}) take the same path (argued from the code, not run).
> h3. Impact
> * Each update of a non empty directory key adds the directory's file count and size again to its parent and every ancestor up to the bucket, and so to the volume and root totals, which the endpoints compute by adding up the bucket summaries. In the reproduction one {{addAcl}} turned a bucket with 1 file and 100 bytes into 2 files and 200 bytes.
> * The extra amount is never subtracted: later deletes subtract only the real content. It is served by the namespace summary and disk usage endpoints until a full NSSummary rebuild. OM data is not affected.
> * Since when: the parent id has been read this way since HDDS-7121, but a missed subtraction only became a double count when HDDS-13432 made directory updates carry totals up the tree, so 2.1.0 and later are affected.
> h3. Reproduction
> Deterministic, unmodified source. [^TestBugCR3LegacyDirUpdateDoubleCount.java] ({{hadoop-ozone/integration-test-recon}}) starts a mini cluster with Recon and filesystem paths enabled and uses only public client calls: {{createDirectory("dir1")}}, a 100 byte {{dir1/file1}}, a Recon sync, {{addAcl}} on {{dir1/}}, a 10 byte {{marker}} key, and another sync. The marker shows when Recon has processed the sync with the ACL change. On the unfixed code it fails with {{Bucket after ACL change on dir1/ and the marker key: numOfFiles=3 sizeOfFiles=210 (expected 2 and 110)}}. With the patch it passes.
> h3. Patch
> [^CR-3-resolve-old-parent-of-legacy-directory-update.patch], against 0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at 2264d4d48ac.
> The old directory gets the parent already resolved for the new value. For an UPDATE both values are stored under the same key and have the same key name, so the parent is the same, and no second lookup is needed. Covered by the new {{testProcessDirKeyUpdateKeepsAncestorTotals}} in the existing {{TestNSSummaryTaskWithLegacy}} (an UPDATE on {{dir1/dir2/}} must leave {{dir1}} and the bucket unchanged), and by {{testProcessUpdateFileSize}}, which asserted the double count (2 files in a bucket that holds 1) and now expects 1. Without the change both fail. With it the existing Recon NSSummary, endpoint and event handler suites pass and checkstyle is clean.
> Found by code review of the Recon namespace summary and task sync paths, as part of the TLA+ verification effort under HDDS-15926, on commit 0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org