[jira] [Updated] (HDDS-16145) Checkpoint flush during delete transaction removal can double count the SCM deleted block summary

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated HDDS-16145:
----------------------------------
    Labels: pull-request-available  (was: )

> Checkpoint flush during delete transaction removal can double count the SCM deleted block summary
> -------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16145
>                 URL: https://issues.apache.org/jira/browse/HDDS-16145
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Assignee: Priyesh Karatha
>            Priority: Major
>              Labels: pull-request-available
>
> A checkpoint or leader-transfer flush that interposes in the middle of {{DeletedBlockLogStateManagerImpl.removeTransactionsFromDB}} can re-expose a delete-transaction row that is still durable, causing the block-deletion scanner to select it again and decrement the volatile summary a second time, to a permanent and impossible negative value.
> h3. Steps to reproduce
> # The block-deletion scanner drives terminal cleanup, which calls {{removeTransactionsFromDB(txIDs, summary)}}. That method first marks the ids in {{deletingTxIDs}} (the scanner-hide set), then removes each row from the transaction buffer, then buffers the updated summary, as separate steps.
> # A checkpoint download ({{InterSCMGrpcService}}) or leader-transfer flush runs {{transactionBuffer.flush()}} between the mark and the buffered removal. {{onFlush}} then resets {{deletingTxIDs}} to a fresh empty set, clearing the hide marker while the row is still durable (its {{removeFromBuffer}} had not been flushed).
> # The next scan no longer sees the row as hidden, re-selects it, and terminal cleanup decrements the summary again for the same transaction.
> The summary is driven to a negative value (for a single transaction it reaches -1), which is persisted durably and reloaded verbatim on restart and leader transfer.
> h3. Root cause
> {{removeTransactionsFromDB}} performs the hide marking ({{deletingTxIDs.addAll}}), the row removal ({{removeFromBuffer}}), and the summary update as separate, non-atomic steps. {{SCMHADBTransactionBufferImpl.flush()}} takes only the write lock and does not consult the in-progress apply/removal state, and {{onFlush}} resets {{deletingTxIDs}}. A flush interposed mid-removal therefore clears the hide marker for a row that is still durable, so the scanner re-selects it and double-decrements the summary. A negative deleted-block count is an impossible undercount.
> h3. Impact
> The deleted-block transaction summary is observability only. It is surfaced by the SCM client protocol {{getDeletedBlockSummary}}, {{ozone admin scm}}, Recon {{PendingDeletionEndpoint}}, and JMX metrics, and does not affect deletion correctness. The consequence is a persistent, operator visible accounting corruption (a negative pending-deletion count) that survives restart and leader transfer. There is no data loss and no premature deletion.
> h3. Relation to HDDS-15665
> HDDS-15665 stabilizes an already-open deleted-block iterator against concurrent modification, but does not make {{removeTransactionsFromDB}} atomic with respect to {{flush()}}, nor does it protect a new iterator opened after a mid-removal flush clears the hide marker. No filed issue reports this checkpoint-flush re-exposure and summary double decrement.
> h3. Suggested fix
> Make the hide marking, row removal, and summary update in {{removeTransactionsFromDB}} atomic with respect to flush, or have {{flush()}}/{{onFlush}} preserve the {{deletingTxIDs}} hide markers for rows whose removal has not yet been flushed, so a flush cannot re-expose a still-durable row that is being removed. Recomputing the summary from the durable rows on reload would also prevent a negative value from being persisted.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org