You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Krishna Kumar Asawa (Jira)" <ji...@apache.org> on 2026/10/19 09:27:00 UTC
[jira] [Assigned] (HDDS-17043) Recon emits a second DELETE event with a stale value when a key is deleted twice within one delta sync, and subtracts the key twice
[ https://issues.apache.org/jira/browse/HDDS-17043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Krishna Kumar Asawa reassigned HDDS-17043:
------------------------------------------
Assignee: Arafat Khan
> Recon emits a second DELETE event with a stale value when a key is deleted twice within one delta sync, and subtracts the key twice
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17043
> URL: https://issues.apache.org/jira/browse/HDDS-17043
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Arafat Khan
> Priority: Minor
> Attachments: MC-4-drop-repeated-delete-event-in-one-recon-delta.patch, TestBugMC4DoubleDeleteInOneDelta.java
>
>
> h3. Mechanism
> {{OMDBUpdatesHandler.processEvent}} takes the old value of a key from the latest event for that key in the current Recon delta when there is one, and from the Recon OM DB otherwise. After a DELETE event that latest value is the row that was deleted. The DELETE branch drops an event when the old value is null, so a second DELETE of the same key in the same delta is emitted again with the stale row, and {{NSSummaryTaskDbEventHandler.handleDeleteKeyEvent}} subtracts that key a second time. Without the earlier event the old value would be read from the Recon OM DB, where the key is already gone, and the event would be dropped.
> OM produces this WAL shape in normal operation: {{OzoneManagerDoubleBuffer}} writes one flush as one batch, and {{RDBBatchOperation}} keeps only the last write per key, so a key that is committed and deleted again within one flush reaches the WAL as a bare DELETE.
> h3. Trigger
> # Key {{k}} exists when Recon last synced.
> # OM flush 1: {{k}} is deleted.
> # OM flush 2: {{k}} is created again (commit) and deleted again, both applied before the same double buffer flush.
> # One Recon delta sync covers both flushes.
> No fault is needed. Whether both deletes fall into one Recon delta and the re-create and delete into one OM flush depends on timing and load; I did not run this on a cluster.
> h3. Impact
> * The bucket (or the FSO parent and its ancestors) loses the key's file count and size twice. In the reproduction the bucket ended at 0 files and {{-50}} bytes instead of 1 file and 50 bytes.
> * The wrong totals are served by the namespace summary and disk usage endpoints and are kept by every later incremental sync until a full NSSummary rebuild. OM data is not affected.
> * Every Recon OM task receives the duplicate DELETE and subtracts it again: {{FileSizeCountTask}} decrements the file size bin of the stale value again, with no floor; {{OmTableInsightTask}} decrements the table's object count again (and, for the open key and deleted tables, the sizes); {{ContainerKeyMapperTask}} still finds the key's container rows, which are removed only when the batch is written, and decrements each container's key count again (argued from the code, not run).
> * Since when: OM writes a re-create and delete of a key in one batch as a bare DELETE since HDDS-8128, so 1.4.0 and later are affected.
> h3. Reproduction
> Deterministic, unmodified source. [^TestBugMC4DoubleDeleteInOneDelta.java] ({{hadoop-ozone/recon}}) applies real {{OMKeyCommitResponse}} and {{OMKeyDeleteResponse}} objects to a real OM DB, one batch per flush, reads the delta with {{RDBStore.getUpdatesSince}}, iterates each WAL batch into {{OMDBUpdatesHandler}} and applies it to the Recon OM DB as {{OzoneManagerServiceProviderImpl}} does, and runs {{NSSummaryTaskWithOBS}}. The expected totals come from a full rebuild of the same Recon OM DB. With the re-create and second delete in one flush, the handler emits two {{DELETE /vol1/ob1/k}} events with the same stale value (objectID 2002, 100 bytes) and the test fails with {{numOfFiles incremental vs rebuild ==> expected: <1> but was: <0>}}. The second test, with the re-create and second delete in separate flushes, passes. With the patch both pass.
> h3. Patch
> [^MC-4-drop-repeated-delete-event-in-one-recon-delta.patch], against 0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at 2264d4d48ac.
> In the DELETE branch of {{OMDBUpdatesHandler.processEvent}}, a DELETE whose latest event in this delta is already a DELETE produces no event, as for any DELETE of a key Recon does not have. Later events for the key are unchanged. {{TestOMDBUpdatesHandler.testOperateOnSameEntry}} asserted the duplicate DELETE (7 events); it now expects 6 and keeps its other assertions. Without the change it fails with {{expected: <6> but was: <7>}}. With it the existing Recon task and handler suites pass and checkstyle is clean.
> Found by TLA+ model checking and code review of the Recon namespace summary and task sync paths under HDDS-15926, on commit 0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org