[jira] [Created] (HDDS-16144) SCM may purge a delete transaction before a newly copied replica has acknowledged it

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16144:
---------------------------------

             Summary: SCM may purge a delete transaction before a newly copied replica has acknowledged it
                 Key: HDDS-16144
                 URL: https://issues.apache.org/jira/browse/HDDS-16144
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng


{{SCMDeletedBlockTransactionStatusManager.commitTransactions}} decides whether a delete transaction can be purged from the deleted-block log using a replica set snapshot taken earlier in the same call, and then removes the transaction from the durable log without re-checking membership. A replica that is copied to a new datanode after the snapshot but before the purge, and that never received or acknowledged the deletion, can therefore keep the blocks forever: SCM has already purged the transaction and never re-issues it.

h3. Steps to reproduce

# A delete transaction for a container is outstanding and has been sent to the current replicas.
# {{commitTransactions}} captures the replica set via {{containerManager.getContainerReplicas(containerId)}} and evaluates {{canPurge}} from that snapshot (all snapshot replicas have committed).
# Around the same time, replication manager copies the container to a new datanode. The copy carries the container RocksDB, so the new replica inherits whatever delete state the source had; if the source had not yet physically deleted the blocks, the new replica holds them with no pending SCM delete transaction.
# {{commitTransactions}} proceeds to purge: {{transactionToDNsCommitMap.remove(txID)}} and {{removeTransactions(txIDsToBeDeleted)}} delete the row from the durable deleted-block log. There is no re-validation of the current replica set at removal time.

The transaction is gone from the SCM log and is never recreated. SCM does not reconcile a replica reported {{deleteTransactionId}} against the container and does not re-issue delete transactions from it (the field is effectively unused in server-scm), so the lagging replica is never told to delete.

h3. Root cause

The purge decision and the durable removal are not guarded by a consistent, current replica set. {{commitTransactions}} evaluates {{canPurge}} against a replica snapshot and then {{removeTransactions}} removes the row with no membership re-check. SCM has no catch-up mechanism that compares a replica's {{deleteTransactionId}} to the container's and re-issues missed deletions; once the transaction is purged it is never re-delivered.

h3. Impact

A container replica that becomes current in the snapshot-to-purge window without having applied the deletion permanently retains the deleted blocks, an undeleted-block space leak on that datanode with no automatic recovery at the SCM level. The leak is confined to datanode capacity; there is no client visible data corruption. The SCM-level premature purge reproduces deterministically; demonstrating the physical on-disk leak requires a mini-cluster with adversarial copy-versus-delete timing, so this is filed as an environment-limited (not fully cluster-reproduced) finding.

h3. Relation to HDDS-11498

HDDS-11498 improved deletion efficiency and the "send to all replicas or not at all" handling in {{commitTransactions}}, but does not address a replica that becomes current after the ACK-set snapshot is taken and before the durable removal. No filed issue reports this stale-snapshot premature purge.

h3. Suggested fix

Re-validate the current replica set immediately before removing a transaction from the durable log (recompute {{canPurge}} against a fresh {{getContainerReplicas}} under the same critical section), or reconcile a newly added replica's {{deleteTransactionId}} against the container and re-issue any missed delete transactions before allowing the purge, so a replica that has not acknowledged the deletion cannot be orphaned.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org