You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2026/04/14 03:05:00 UTC
[jira] [Assigned] (HDDS-15027) [Docs] [System Internals] Implementation of Delete Operations
[ https://issues.apache.org/jira/browse/HDDS-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei-Chiu Chuang reassigned HDDS-15027:
--------------------------------------
Assignee: Wei-Chiu Chuang
> [Docs] [System Internals] Implementation of Delete Operations
> -------------------------------------------------------------
>
> Key: HDDS-15027
> URL: https://issues.apache.org/jira/browse/HDDS-15027
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: documentation
> Reporter: Wei-Chiu Chuang
> Assignee: Wei-Chiu Chuang
> Priority: Major
> Labels: pull-request-available
>
> A very common question users ask is: when a user issues a delete request, how does it eventually get deleted? Due to Ozone's distributed nature, it can be tricky to understand all the moving parts, so this page gives a holistic overview of delete.
> ---
> Client side:
> ✦ Client-side deletion in Ozone depends on the interface and bucket layout, with trash operations implemented as a rename to .Trash/Current by both the
> Ozone CLI and FileSystem.
> 1. Ozone CLI Deletion (DeleteKeyHandler)
> * FSO Buckets:
> * If Trash is enabled (fs.trash.interval > 0), the CLI renames the key to .Trash/<user>/Current/<keyName>.
> * If Trash is disabled, it calls bucket.deleteKey(keyName) for direct deletion.
> * OBS/Legacy Buckets:
> * Directly calls bucket.deleteKey(keyName), which triggers a logical delete in OM. Trash is not natively supported for non-FSO buckets via the ozone
> sh key delete command.
> 2. Ozone FileSystem Deletion (OzoneFileSystem)
> The FileSystem interface (o3fs:// or ofs://) typically uses the standard Hadoop Trash Policy:
> * When you run hadoop fs -rm <path>, the Hadoop client code checks the configuration.
> * If trash is enabled, the client performs a Rename operation:
> * Source: /path/to/file
> * Destination: /.Trash/Current/path/to/file
> * If -skipTrash is used or trash is disabled, it calls FileSystem.delete(), which maps to a direct deleteKey RPC to the Ozone Manager.
> 3. OM Logic for Deletion vs. Trash
> * Rename (Trash): To the OM, moving a file to trash is just a metadata update in the keyTable or fileTable. The object ID remains the same, and no blocks
> are reclaimed.
> * Direct Delete: Triggered by bucket.deleteKey(). As previously explained, OM moves the metadata to the deletedTable, starting the asynchronous block
> reclamation process.
> 4. Trash Emptier
> Because trash is just a normal directory (.Trash), it is emptied by a Trash Emptier (usually a background thread in the Hadoop client or a scheduled
> service):
> 1. It renames .Trash/Current to a timestamped checkpoint like .Trash/<timestamp>.
> 2. After the retention period expires, it issues a recursive delete on the timestamped directory.
> 3. This recursive delete finally moves the keys to the OM's deletedTable for physical reclamation.
> Summary Flow
> ┌───────────────┬──────────────┬───────────────────────┬─────────────────────────────────────────────────────────┐
> │ Client Action │ Trash Status │ OM Operation │ Result │
> ├───────────────┼──────────────┼───────────────────────┼─────────────────────────────────────────────────────────┤
> │ rm <file> │ Disabled │ deleteKey │ Moved to deletedTable (Asynchronous reclamation begins) │
> │ rm <file> │ Enabled │ renameKey │ Moved to .Trash/Current (No space reclaimed yet) │
> │ rm -skipTrash │ Enabled │ deleteKey │ Moved to deletedTable (Immediate reclamation begins) │
> │ Trash Emptier │ N/A │ deleteKey (recursive) │ Moved to deletedTable (Space finally reclaimed) │
> └───────────────┴──────────────┴───────────────────────┴─────────────────────────────────────────────────────────┘
> Key takeaway: Trash in Ozone is a Client-Side simulation achieved via renames. The Ozone Manager itself does not have a "Trash Table"; it only sees
> renames or final deletes.
> ---
> Inside OM:
> ✦ The deletion process in Apache Ozone is a multi-phase, asynchronous workflow designed to handle massive scale and File System Optimized (FSO) structures
> efficiently.
> Based on the provided source files, here is the lifecycle of a deleted file or directory as it moves through the Ozone Manager (OM) tables:
> 1. Initial Deletion (Synchronous Phase)
> When a client issues a delete request, OMKeyDeleteRequest handles the transaction:
> * For Files: The entry is removed from the fileTable (or keyTable in non-FSO buckets) and a new entry is added to the deletedTable.
> * For Directories (FSO): The directory is removed from the directoryTable and moved to the deletedDirectoryTable.
> * Quota Update: The bucket's used bytes and namespace quota are updated at this stage to reflect the "logical" deletion.
> 2. Directory Expansion (DirectoryDeletingService)
> For FSO buckets, deleting a directory does not immediately move all its children. Instead, the DirectoryDeletingService processes them asynchronously:
> * Scanning: It scans the deletedDirectoryTable for pending deletions.
> * Recursive Movement:
> * Sub-files: It finds all files within the deleted directory and moves them from the fileTable to the deletedTable.
> * Sub-directories: It finds sub-directories and moves them from the directoryTable to the deletedDirectoryTable so they can be processed in the next
> iteration.
> * Purging the Directory: Once a directory is empty (all children have been moved to the deleted tables), the service sends a PurgePathRequest to remove
> the directory itself from the deletedDirectoryTable.
> 3. Block Reclamation (KeyDeletingService)
> The KeyDeletingService is responsible for the physical reclamation of storage space occupied by files in the deletedTable:
> * SCM Coordination: It scans the deletedTable, extracts the block list (BlockGroups) for each file, and sends a request to the Storage Container Manager
> (SCM) to delete these blocks.
> * Wait for ACK: It waits for SCM to acknowledge that the block deletion intent has been persisted.
> * Trigger Purge: Once SCM acknowledges, the service initiates the final cleanup by submitting an OMKeyPurgeRequest.
> 4. Final Metadata Cleanup (OMKeyPurgeRequest)
> This is the terminal state for a file's metadata:
> * Purging: The OMKeyPurgeRequest removes the entries from the deletedTable.
> * Snapshot Handling: If snapshots are involved, it may move keys to a "previous" snapshot's deleted table instead of purging them, ensuring data
> consistency across the snapshot chain.
> Summary Table Flow
> ┌──────┬────────────┬───────────────────────┬───────────────────────┬──────────────────────────────────────────────┐
> │ Step │ Entity │ Source Table │ Destination Table │ Handled By │
> ├──────┼────────────┼───────────────────────┼───────────────────────┼──────────────────────────────────────────────┤
> │ 1 │ File │ fileTable │ deletedTable │ OMKeyDeleteRequest │
> │ 1 │ Directory │ directoryTable │ deletedDirectoryTable │ OMKeyDeleteRequest (FSO) │
> │ 2 │ Child File │ fileTable │ deletedTable │ DirectoryDeletingService │
> │ 2 │ Child Dir │ directoryTable │ deletedDirectoryTable │ DirectoryDeletingService │
> │ 3 │ Final File │ deletedTable │ Removed │ KeyDeletingService -> OMKeyPurgeRequest │
> │ 4 │ Final Dir │ deletedDirectoryTable │ Removed │ DirectoryDeletingService -> PurgePathRequest │
> └──────┴────────────┴───────────────────────┴───────────────────────┴──────────────────────────────────────────────┘
> ---
> SCM:
> ✦ The block reclamation process in SCM (Storage Container Manager) is an asynchronous, transaction-based workflow that ensures data is physically deleted
> from Datanodes while maintaining consistency across replicas.
> Here is the step-by-step implementation of how SCM handles block reclamation:
> 1. Transaction Persistence (DeletedBlockLog)
> When SCM receives a deleteKeyBlocks request from the Ozone Manager (OM), it doesn't delete the blocks immediately. Instead, it creates a durable record:
> * Grouping: SCM groups the blocks by their Container ID.
> * Logging: It creates a DeletedBlocksTransaction for each container. Each transaction is assigned a unique Transaction ID (TXID).
> * Storage: These transactions are persisted in the SCM metadata database in the DeletedBlocksTXTable. This ensures that even if SCM restarts, it won't
> "forget" to delete these blocks.
> 2. Throttled Command Dispatch (SCMBlockDeletingService)
> A background service called SCMBlockDeletingService periodically scans the DeletedBlocksTXTable to move the process forward:
> * Node Selection: It identifies healthy Datanodes that have room in their command queues (to avoid overwhelming them).
> * Transaction Matching: For each pending transaction, it identifies which Datanodes hold a replica of that container.
> * Command Creation: It bundles multiple transactions into a DeleteBlocksCommand for a specific Datanode.
> * Dispatch: The command is sent to the Datanode, typically as a payload in a heartbeat response.
> 3. Physical Deletion (Datanode Side)
> Once a Datanode receives the DeleteBlocksCommand:
> * It iterates through the transactions and physically deletes the chunks associated with the listed Block IDs from its local disks.
> * It records the success or failure of each transaction.
> 4. Acknowledgement and Final Cleanup
> SCM needs to ensure that all replicas of a block are deleted before it considers the task complete:
> * ACK Reporting: The Datanode sends back a ContainerBlocksDeletionACKProto (via DeleteBlockStatus report) to SCM.
> * Tracking: SCM’s SCMDeletedBlockTransactionStatusManager tracks which Datanodes have acknowledged which TXIDs.
> * Removal: Only after every replica (e.g., all 3 nodes in a Ratis pipeline) has successfully acknowledged the deletion does SCM finally remove the
> transaction from the DeletedBlocksTXTable.
> Why this architecture?
> * Fault Tolerance: By using a persistent log, SCM ensures that deletions happen eventually even if nodes are temporarily down.
> * Consistency: SCM won't stop retrying until every single replica is confirmed deleted, preventing "orphan" blocks from wasting space.
> * Performance: Batching multiple block deletions into a single transaction and command reduces RPC overhead between OM, SCM, and Datanodes.
> ---
> Datanode:
> ✦ Block deletion in the Datanode is a two-step process: first, it marks the metadata for deletion upon receiving a command from SCM, and then a background
> service physically removes the data from the disk.
> Here is the detailed implementation:
> 1. Metadata Marking (DeleteBlocksCommandHandler)
> When the Datanode receives a DeleteBlocksCommand from SCM, it triggers the DeleteBlocksCommandHandler:
> * Queueing: The command is added to a deleteCommandQueues. A dedicated DeleteCmdWorker thread picks up the commands.
> * Transaction Processing: For each transaction in the command, a ProcessTransactionTask is executed.
> * Metadata Update:
> * Schema V2/V3: It adds the transaction to a DeletedBlocksTXTable within the container's database.
> * Schema V1: It moves the block entry from the blockDataTable to the deletedBlocksTable (effectively renaming the key to a "deleting" prefix).
> * ACK: Once the metadata is updated, the Datanode sends an Acknowledgement (ACK) back to SCM. SCM then considers its part of the transaction "done" (once
> all replicas ACK).
> 2. Physical Deletion (BlockDeletingService)
> The actual removal of chunks from the disk is handled by a background service to avoid blocking the main heartbeat/command processing:
> * Choosing Containers: The BlockDeletingService periodically scans containers and identifies those with pending deletions.
> * Task Execution: It creates a BlockDeletingTask for each container.
> * Chunk Removal:
> * The task iterates through the DeletedBlocksTXTable (or the deleted blocks in Schema V1).
> * It calls handler.deleteBlock(), which physically deletes the chunk files from the local file system.
> * DB Cleanup: After the files are deleted, the service:
> * Removes the block entries from the blockDataTable.
> * Removes the transaction from the DeletedBlocksTXTable.
> * Updates the container's metadata (e.g., reduces usedBytes and pendingDeletionBlocks count).
> * Empty Container Check: If a container becomes completely empty after deletion, it is marked as such, allowing SCM to potentially delete the container
> itself later.
> Summary of Component Roles
> ┌────────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────┐
> │ Component │ Responsibility │
> ├────────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────┤
> │ DeleteBlocksCommandHandler │ Receives RPC from SCM, persists deletion intent in the container DB, and sends ACK. │
> │ BlockDeletingService │ Background thread that throttles and schedules physical disk cleanup tasks. │
> │ BlockDeletingTask │ Performs the actual rm of chunk files and updates the container DB counters. │
> │ Handler │ Interface used by the task to interact with the specific storage engine (e.g., KeyValueHandler). │
> └────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────┘
> This decoupled architecture ensures that the Ozone Manager and SCM don't have to wait for slow disk I/O to confirm a deletion, while still guaranteeing
> that the space will eventually be reclaimed.
> --
> In the case of OBS buckets:
> ✦ OBS and LEGACY buckets use a flat key structure, simplifying the deletion process in the Ozone Manager. Unlike FSO buckets, which require
> DirectoryDeletingService to expand directory deletions, OBS buckets move keys directly from the keyTable to the deletedTable.
> 1. Synchronous Phase (OMKeyDeleteRequest)
> When a client deletes an object in an OBS bucket:
> * Table Access: OM accesses the keyTable (specifically getKeyTable(BucketLayout.OBJECT_STORE)).
> * Tombstone: It adds a cache entry (tombstone) to the keyTable to mark the key as deleted in memory.
> * Quota: The bucket's usedBytes and usedNamespace are updated immediately.
> 2. Response and Batch Update (OMKeyDeleteResponse)
> The addToDBBatch method in the response handles the persistent changes:
> * Remove from Source: The key is deleted from the keyTable.
> * Move to Deleted: The key's metadata (OmKeyInfo) is wrapped into a RepeatedOmKeyInfo and added to the deletedTable.
> * Note: Even though OBS has "directories" in the name (e.g., dir1/file1), these are just string prefixes. There is no directoryTable or
> deletedDirectoryTable involved.
> * Deleted Table Key Format: The key in the deletedTable is typically formatted as volume/bucket/keyName/objectID.
> 3. Asynchronous Reclamation (KeyDeletingService)
> From this point forward, the process is identical to the FSO file deletion process:
> * KeyDeletingService picks up the entry from the deletedTable.
> * It coordinates with SCM to delete the physical blocks.
> * Once SCM acknowledges, the entry is permanently removed from the deletedTable via OMKeyPurgeRequest.
> Comparison: FSO vs. OBS Deletion in OM
> ┌──────────────────────┬─────────────────────────────────────┬─────────────────────┐
> │ Feature │ FSO Bucket │ OBS / LEGACY Bucket │
> ├──────────────────────┼─────────────────────────────────────┼─────────────────────┤
> │ Source Tables │ fileTable, directoryTable │ keyTable │
> │ Intermediate Table │ deletedDirectoryTable (for dirs) │ None │
> │ Final Deletion Table │ deletedTable (for files) │ deletedTable │
> │ Background Expansion │ Required (DirectoryDeletingService) │ Not Required │
> │ Complexity │ High (Tree traversal needed) │ Low (Direct move) │
> └──────────────────────┴─────────────────────────────────────┴─────────────────────┘
> In short, OBS deletion is much simpler because it bypasses the entire "directory expansion" logic. Every object is treated as a single independent unit,
> regardless of whether its name contains slashes.
> ---
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org