[jira] [Assigned] (HDDS-16144) SCM may purge a delete transaction before a newly copied replica has acknowledged it

["Priyesh Karatha (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Priyesh Karatha reassigned HDDS-16144:
--------------------------------------

    Assignee: Priyesh Karatha

> SCM may purge a delete transaction before a newly copied replica has acknowledged it
> ------------------------------------------------------------------------------------
>
>                 Key: HDDS-16144
>                 URL: https://issues.apache.org/jira/browse/HDDS-16144
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Assignee: Priyesh Karatha
>            Priority: Major
>         Attachments: HDDS-16144.001.patch
>
>
> {{SCMDeletedBlockTransactionStatusManager.commitTransactions}} decides whether a delete transaction can be purged from the deleted-block log using a replica set snapshot taken earlier in the same call, and then removes the transaction from the durable log without re-checking membership. A replica that is copied to a new datanode after the snapshot but before the purge, and that never received or acknowledged the deletion, can therefore keep the blocks forever: SCM has already purged the transaction and never re-issues it.
> h3. Steps to reproduce
> # A delete transaction for a container is outstanding and has been sent to the current replicas.
> # {{commitTransactions}} captures the replica set via {{containerManager.getContainerReplicas(containerId)}} and evaluates {{canPurge}} from that snapshot (all snapshot replicas have committed).
> # Around the same time, replication manager copies the container to a new datanode. The copy carries the container RocksDB, so the new replica inherits whatever delete state the source had; if the source had not yet physically deleted the blocks, the new replica holds them with no pending SCM delete transaction.
> # {{commitTransactions}} proceeds to purge: {{transactionToDNsCommitMap.remove(txID)}} and {{removeTransactions(txIDsToBeDeleted)}} delete the row from the durable deleted-block log. There is no re-validation of the current replica set at removal time.
> The transaction is gone from the SCM log and is never recreated. SCM does not reconcile a replica reported {{deleteTransactionId}} against the container and does not re-issue delete transactions from it (the field is effectively unused in server-scm), so the lagging replica is never told to delete.
> h3. Root cause
> The purge decision and the durable removal are not guarded by a consistent, current replica set. {{commitTransactions}} evaluates {{canPurge}} against a replica snapshot and then {{removeTransactions}} removes the row with no membership re-check. SCM has no catch-up mechanism that compares a replica's {{deleteTransactionId}} to the container's and re-issues missed deletions; once the transaction is purged it is never re-delivered.
> h3. Impact
> A container replica that becomes current in the snapshot-to-purge window without having applied the deletion permanently retains the deleted blocks, an undeleted-block space leak on that datanode with no automatic recovery at the SCM level. The leak is confined to datanode capacity; there is no client visible data corruption. The SCM-level premature purge reproduces deterministically; demonstrating the physical on-disk leak requires a mini-cluster with adversarial copy-versus-delete timing, so this is filed as an environment-limited (not fully cluster-reproduced) finding.
> h3. Relation to HDDS-11498
> HDDS-11498 improved deletion efficiency and the "send to all replicas or not at all" handling in {{commitTransactions}}, but does not address a replica that becomes current after the ACK-set snapshot is taken and before the durable removal. No filed issue reports this stale-snapshot premature purge.
> h3. Suggested fix
> Re-validate the current replica set immediately before removing a transaction from the durable log (recompute {{canPurge}} against a fresh {{getContainerReplicas}} under the same critical section), or reconcile a newly added replica's {{deleteTransactionId}} against the container and re-issue any missed delete transactions before allowing the purge, so a replica that has not acknowledged the deletion cannot be orphaned.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org