You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Devesh Kumar Singh (Jira)" <ji...@apache.org> on 2026/10/16 06:28:00 UTC

[jira] [Created] (HDDS-17059) Implement Jobworker-based key migration to satisfy storage policies

Devesh Kumar Singh created HDDS-17059:
-----------------------------------------

             Summary: Implement Jobworker-based key migration to satisfy storage policies
                 Key: HDDS-17059
                 URL: https://issues.apache.org/jira/browse/HDDS-17059
             Project: Apache Ozone
          Issue Type: Sub-task
          Components: Ozone Datanode, Ozone Filesystem, Ozone Manager
            Reporter: Devesh Kumar Singh
            Assignee: Devesh Kumar Singh


Updating a bucket’s storage policy changes the desired placement for applicable writes, but existing keys remain on their current storage volumes. For example, changing a bucket from HOT to COLD does not move existing RATIS THREE keys from SSD to ARCHIVE.

Introduce asynchronous key migration using Jobworker to rewrite existing keys onto the target storage tier while preserving their replication configuration.

Bucket-policy updates through CLI or native APIs and migration submission are separate operations; updating the policy alone must not be represented as completed data migration.

*Acceptance criteria:*

- A RATIS THREE key can migrate from SSD to ARCHIVE while retaining its name, content, replication configuration, and required attributes.
- Placement respects the target policy and configured fallback rules.
- Failed or incomplete copies cannot replace the committed source key.
- Concurrent overwrites or deletions cannot be overwritten by stale migration work.
- Tasks recover across OM failover and Jobworker restart without losing work or double-counting results.
- Progress, failures, retries, and cancellation outcomes are observable.
- Integration tests verify actual DN placement, data correctness, failure recovery, and unchanged replication.

The migration flow should:

1. Persist migration tasks and batches of key names, observed generations, and target storage policies in OM.
2. Allow lifecycle processing to create migration work.
3. Use StoragePolicySatisfierService to schedule tasks, select healthy Jobworkers, and dispatch commands through heartbeat responses.
4. Have Jobworker read each source key and invoke policy-aware rewriteKey, retaining the source replication configuration.
5. Reuse the OM → SCM → datanode write path to allocate and write target-tier blocks.
6. Validate the source generation before committing the replacement, protecting concurrent user updates.
7. Report per-key outcomes, retry eligible failures, and persist task progress through OM Ratis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org