You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Devesh Kumar Singh (Jira)" <ji...@apache.org> on 2026/10/16 12:06:00 UTC
[jira] [Updated] (HDDS-17059) Introduce a generic Jobworker framework for OM-managed background jobs.
[ https://issues.apache.org/jira/browse/HDDS-17059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devesh Kumar Singh updated HDDS-17059:
--------------------------------------
Summary: Introduce a generic Jobworker framework for OM-managed background jobs. (was: Implement Jobworker-based key migration to satisfy storage policies)
> Introduce a generic Jobworker framework for OM-managed background jobs.
> -----------------------------------------------------------------------
>
> Key: HDDS-17059
> URL: https://issues.apache.org/jira/browse/HDDS-17059
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode, Ozone Filesystem, Ozone Manager
> Reporter: Devesh Kumar Singh
> Assignee: Devesh Kumar Singh
> Priority: Major
>
> Updating a bucket’s storage policy changes the desired placement for applicable writes, but existing keys remain on their current storage volumes. For example, changing a bucket from HOT to COLD does not move existing RATIS THREE keys from SSD to ARCHIVE.
> Introduce asynchronous key migration using Jobworker to rewrite existing keys onto the target storage tier while preserving their replication configuration.
> Bucket-policy updates through CLI or native APIs and migration submission are separate operations; updating the policy alone must not be represented as completed data migration.
> *Acceptance criteria:*
> - A RATIS THREE key can migrate from SSD to ARCHIVE while retaining its name, content, replication configuration, and required attributes.
> - Placement respects the target policy and configured fallback rules.
> - Failed or incomplete copies cannot replace the committed source key.
> - Concurrent overwrites or deletions cannot be overwritten by stale migration work.
> - Tasks recover across OM failover and Jobworker restart without losing work or double-counting results.
> - Progress, failures, retries, and cancellation outcomes are observable.
> - Integration tests verify actual DN placement, data correctness, failure recovery, and unchanged replication.
> The migration flow should:
> 1. Persist migration tasks and batches of key names, observed generations, and target storage policies in OM.
> 2. Allow lifecycle processing to create migration work.
> 3. Use StoragePolicySatisfierService to schedule tasks, select healthy Jobworkers, and dispatch commands through heartbeat responses.
> 4. Have Jobworker read each source key and invoke policy-aware rewriteKey, retaining the source replication configuration.
> 5. Reuse the OM → SCM → datanode write path to allocate and write target-tier blocks.
> 6. Validate the source generation before committing the replacement, protecting concurrent user updates.
> 7. Report per-key outcomes, retry eligible failures, and persist task progress through OM Ratis.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org