[jira] [Updated] (HDDS-16143) SCM leader transfer flush can split a delete transaction apply and persist a summary undercount

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16143:
------------------------------
    Attachment: HDDS-16143.001.patch

> SCM leader transfer flush can split a delete transaction apply and persist a summary undercount
> -----------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16143
>                 URL: https://issues.apache.org/jira/browse/HDDS-16143
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: HDDS-16143.001.patch
>
>
> When SCM leadership changes, {{SCMStateMachine.notifyLeaderChanged}} calls {{transactionBuffer.flush()}} unconditionally. If that flush interposes in the middle of a delete-transaction apply, it can persist the transaction rows without the matching summary update, and the subsequent leader-side reconstruction reads the resulting durable summary undercount into the in-memory summary, baking in a permanent undercount that survives restart.
> h3. Steps to reproduce
> # A delete-transaction apply is in progress. {{DeletedBlockLogStateManagerImpl.addTransactionsToDB}} buffers the transaction rows first and then buffers the updated summary as a separate {{addToBuffer}} call.
> # An SCM leader transfer fires and {{SCMStateMachine.notifyLeaderChanged}} calls {{transactionBuffer.flush()}} between the row buffering and the summary buffering.
> # The flush persists the rows with the stale (pre-update) summary, so the durable summary is now lower than the durable row count.
> # {{onBecomeLeader}} then runs {{initDataDistributionData}}, which reloads the in-memory summary from that undercounted durable value rather than recomputing it from the rows.
> The reconstructed in-memory summary is permanently short; the next durable write re-persists the undercount, so it survives further leader transfers and restart.
> h3. Root cause
> {{addTransactionsToDB}} performs the row mutation and the summary mutation as two separate buffer operations, so they are not atomic with respect to a flush. The leader-transfer path uses the unguarded {{transactionBuffer.flush()}} in {{notifyLeaderChanged}} (not the {{flushIfNeeded}} variant that skips when an apply is in progress), so a flush can split an in-progress apply. {{onBecomeLeader}} then reconstructs the in-memory summary from the split durable value via {{initDataDistributionData}}, converting a transient durable inconsistency into a permanent in-memory one.
> h3. Impact
> The deleted-block transaction summary is observability only. It is surfaced by the SCM client protocol {{getDeletedBlockSummary}}, {{ozone admin scm}}, Recon, and JMX metrics, and does not affect deletion correctness. The consequence is a persistent, operator visible summary undercount that survives leader transfer and restart. There is no data loss and no premature deletion.
> h3. Relation to HDDS-14572, HDDS-15065
> HDDS-14572 added the leader-transfer summary reconstruction ({{onBecomeLeader}} to {{initDataDistributionData}}) that this bug relies on to make the undercount permanent. HDDS-15065 added the guarded {{flushIfNeeded}} path (which skips flushing while an apply is in progress) but did not guard the {{flush()}} in {{notifyLeaderChanged}}. Neither reports or fixes this split-then-reconstruct undercount.
> h3. Suggested fix
> Make the row-and-summary durable mutation atomic with respect to flush (buffer both under the same guarded section), or guard the leader-transfer {{flush()}} the same way {{flushIfNeeded}} is guarded so it cannot split an in-progress apply, or have {{initDataDistributionData}} recompute the summary from the durable rows rather than trusting a possibly-split persisted summary value.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org