You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "xichen01 (via GitHub)" <gi...@apache.org> on 2024/07/25 15:08:49 UTC
[PR] HDDS-11233. Ozone Storage Policy Support [ozone]
xichen01 opened a new pull request, #6989:
URL: https://github.com/apache/ozone/pull/6989
## What changes were proposed in this pull request?
Design storage policy for for Ozone. Please comment inline on the markdown document to ask questions and post feedback. Switch to `Rich Diff` mode for smoother reading.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11233
## How was this patch tested?
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-3919009991
Hi, do you know if any work is planned for this ticket? AFAIK, the patch diff wasn't added to the branch and is probably out of date right now. What are the next steps here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2875955485
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
Review Comment:
Does LifecycleService/Jobworker have information about all keys that don't comply with the storage policy? Does it do it key by key iteratively?
At the moment, in the patch, I couldn't see any API that allows collecting information about all keys that don't satisfy the storage policy. Also, is there any plan to migrate bulk file migration rather than key by key using `rewriteKey`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3007344267
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
Review Comment:
@chungen0126, I agree about the Job Worker architecture. Still, from an operational perspective, it's often convenient to have a standalone tool/service that runs the actual migration based on the configured team's migration priorities.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3764793577
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
Review Comment:
There is a typo in the png file, "Feedback Tier" -> Fallback Tier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3773095173
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
Review Comment:
Okay, I see why there are cases that some replica's storage type mismatched with container's, due to replicate/migrate/reconstruction fallback. But in this case, when the replica is migrated to the correct storage type volume by Replication Manager, the storage type should be automatically updated by Datanode itself, why we need the RM to send an explicit command to do this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3773018595
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
Review Comment:
> Container replica whose storage type mismatches the container's storage tier.
When will the Container replica's storage type mismatched with the container's storage tier? For the fallback happens during Container allocation, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4378415007
@xichen01 Could you help to take over? My GH account was recently blocked from GH actions (still waiting for support) so no CI can be triggered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3772954987
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
Review Comment:
> A pipeline can support multiple storage tiers.
Is it because that write pipeline is heavy, so a pipeline support have multiple storage tiers, to reduce the overall pipeline count related? There is one thing that if some write error happens on HDD, datanode will close the whole pipeline, which will make the data write requests for both HDD and SSD retry. Retry for HDD might be acceptable, while SSD writes are most latency sensitive, so there could be latency spikes in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3764943896
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
Review Comment:
Will it be better that always allow the fallback, since if the fallback is not allowed, the write will fail. It's better to finish the write on a degraded media, instead of fail the whole write, right?
Or it can be controlled by client, instead of server, since the client/user owners the data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3773258833
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
Review Comment:
RM normally has these situations,
```
Fallback
1. under replication yes
2. topology not satisfied ?
3. decommissioning yes
4. container balancer no
```
I think whether fallback is allowed depends on the case. Data integrity/safety has high priority over storage policy, so if data is under replication or decommissioning a datanode, fallback must be allowed. For container balancer, since it will evaluate utilization based on storage types, I guess it doesn't have fallback requirement. Topology is a question, topology helps to guarantee the data accessibility in case the hardware failure, in this case, it looks like it's priority is higher than storage policy too, so the answer seems yes too.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
Review Comment:
RM normally has these situations,
```
Fallback
1. under replication yes
2. topology not satisfied ?
3. decommissioning yes
4. container balancer no
```
I think whether fallback is allowed depends on the case. Data integrity/safety has high priority over storage policy, so if data is under replication or decommissioning a datanode, fallback must be allowed. For container balancer, since it will evaluate utilization based on storage types, I guess it doesn't have fallback requirement. Topology is a question, topology helps to guarantee the data accessibility in case the hardware failure, in this case, it looks like it's priority is higher than storage policy too, so the answer seems yes too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3851268991
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
Review Comment:
Does “topology mismatch” refer to the MIS_REPLICATED status?
For the MIS_REPLICATED status, the ReplicaManager will add the Container replica (may be a fallback), after which the Container replica status changes to OVER_REPLICATED. At this stage, the ReplicaManager prioritizes deleting replicas with mismatched storage types (If the placement status does not deteriorate).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-2900322703
@xichen01 is there an understanding of the time frame for the functionality to be implemented? I'd start creating the JIRA tickets and implementing them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2867351940
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
+ - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
+ - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
+ - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
+- `ozone admin container info`, supports displaying the storage policy of Containers.
+- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
+- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
+- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
+- Datanode volume related Metrics support displaying the storage policies of volumes.
+- SCM supports displaying the data space information of different storage policies in the cluster.
+
+## Storage Policy Satisfier Service
+
+- Support data migration across different storage policies.
+ - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
+- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
+- Responsible for managing storage policy-related migration work.
+- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+
+## Permissions Management
+
+- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+
+## Storage Policy Supported Replicas Types
+
+- Support replica type storage policies.
+- Support EC type storage policies.
+ - EC types cannot support some storage policies that mix different storage media.
+
+## Storage Policy Supported Bucket Types
+
+- Support all types of buckets.
+
+## Compatible HDFS Commands
+
+HDFS storage policy-related commands are not supported.
+
+Do not support specified storage policy when creating a file using HDFS Filesystem interface, the Hadoop FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy by Create method.
+
+- Ozone needs to record the access frequency of keys/buckets.
+
+## Recon Side Feature
+
+Support displaying relevant storage tier information in Recon. Need to design more detailed. // TODO
+
+# Architecture Design
+
+This section aims to describe the architecture and design concepts of the system without delving into specific implementation details. Its purpose is to validate the feasibility and rationality of the technical route, as well as to assess the workload and technical dependencies of related tasks.
+
+## Writing Key Process
+
+Writing data to the specified tier:
+
+- When a user creates a key, they set the storage policy and fallbackStrategy parameters and send the request to the OM:
+ - CreateKey
+ - CreateFile
+ - CreateMultipartKey
+- Community Ozone:
+ - CreateStreamKey
+ - CreateStreamFile
+ - CreateMultipartStreamKey
+- OM adds the storage policy information to the allocateBlock request sent to the SCM. The actual storage strategy should be calculated based on the storage strategy requirements above.
+- SCM, upon receiving the request, selects the Pipeline based on the storage policy and allocates a Container.
+ - The storage type of each Container replicas should be added to for the Datanodes in the Pipeline, just like the replicaIndex for the EC key.
+ - If the expected storage tier Pipeline cannot be found, attempt to find the fallback tier defined by the storage policy if the fallbackStrategy is allowed. If it cannot be found, the creation process fails.
+ - If the SCM needs to create a new Container for a specific storage tier, SCM needs to record the Container’s storage tier for the Container. So the ContainerData should add a storage tier field.
+- After receiving the Pipeline with the specified storage type information, the client writes the Chunk replica with the storage type information.
+- Datanode writes data to the specified Container replica or creates a Container replica in the specified volume based on the storage type .
+- OMKeyInfo needs to persist in the storage policy field, if the storage policy does not be specified, the storage policy should be null.
+
+## DN Container Replica Creation
+
+- DN should select the appropriate volume based on the storage type.
+- If the Container includes storage type information, DN should store this information in the Container's replicas YAML configuration.
+- When DN starts, it should read the storage type information from the Container's YAML configuration and load it into memory.
+- If the storage type of Container is not specified, it defaults to Disk.
+
+## Storage Policy Management
+
+- Implement key, prefix, and bucket storage policy-related commands as needed, and add related fields to persist tier information.
+
+## SCM Pipeline / ContainerReplica Management and Block Allocation
+
+- Pipeline needs to add tier-related fields to distinguish between different tiers of Pipelines.
+- For Pipelines tier:
+ - Pipeline adds tier attribute, Pipeline belongs to a specific tier
+ - If the Pipeline tier is null, then the Pipeline tier is Disk, so all the Pipeline tier created before storage policy release should be Disk.
+ - After the storage policy is released, the Container which is allocated by the existing Pipeline may not be created normally due to the change of the storage type of the Datanode volume. For example, if all volumes on a Datanode of the existing Pipeline1 are configured as SSD, but the tier of the existing Pipeline1 is Disk and the storage type of the Container replicas is Disk, the creation of this Container will fail because no suitable volume can be found.
+ - For the above scenario the Pipeline should be closed and SCM should create a new Pipeline.
+ - SCM should select the appropriate Pipeline based on the storage policy selection information in the request during getPipelines.
+ - If no suitable Pipeline is found, select the appropriate Datanode to create the Pipeline.
+ - If no suitable Datanode is found, fallback to creating a Pipeline for the defined fallback tier Pipeline.
+ - If a fallback Pipeline tier cannot be created, AllocateBlock fails.
+ - Currently, Pipelines are selected using a polling method (PipelineStateMap#getPipelines). If there are many Pipelines, performance may be affected, so a more efficient method can be considered.
+
+## Datanode Heartbeat Reporting Process
+
+- Datanode reports the storage type information of each volume through heartbeats. SCM aggregates and manages Datanodes based on different storage types, including the remaining capacity of each storage type.
+ - Currently, Datanode reports volume capacity information through NodeReport. The storage type information of volume can be reported to SCM in DATA_VOLUME of getNodeReport.
+ - When creating Pipelines, SCM needs to choose appropriate Datanodes based on storage type information.
+
+## Datanode Storage Type Management
+
+- Referencing HDFS [1], use configuration to define the tier type of each volume. If no tier information is configured for a volume, it defaults to the default tier. (default tier should be DISK)
+ - When Datanode starts, it should load the tier information into memory based on the configuration.
+
+[1] refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section Configuration
+
+## Container Replicas Storage Type Management
+
+- Container replicas need to add storage type related fields and persist storage to support Container migration.
+- ContainerReport should add storage type information (recorded in the Container YAML) of the Container replicas and its located volume storage type (after cluster upgrade, configuration changes, or fallback during migration, the Container storage type may not match its actual storage type).
+ - DiskBalancerService should ensure that the balanced Container resides in the volume of the expected tier during the balancing process.
+
+## SCM Adapts to ReplicationManager
+
+### SCM Adapts to ReplicationManager
+
+- During Container replicas migration, SCM should consider whether the target Datanode contains the corresponding tier and the capacity of that tier.
+ - If the expected storage type cannot be found during migration, attempt to find the storage type defined by fallback storage. If it cannot be found, the migration fails.
+
+### Container replicas Storage Type Satisfier
+
+- Checking whether the Container is in the expected tier. If not, trigger the migration process.
+- Current ReplicationManager logic can be reused, making this check one of its checks.
+- Add a new Container HealthState, such as STORAGE_TYPE_MISMATCH, and support the count the number of the Container in the STORAGE_TYPE_MISMATCH HealthState.
+
+## DN Adapts to Container Migration
+
+- Adapt DiskBalancerService (ReplicationManager) to prioritize migrating Container replicas to their expected storage type.
Review Comment:
Both DiskBalancerService and ReplicationManager need to be adapted.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
+ - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
+ - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
+ - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
+- `ozone admin container info`, supports displaying the storage policy of Containers.
+- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
+- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
+- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
+- Datanode volume related Metrics support displaying the storage policies of volumes.
+- SCM supports displaying the data space information of different storage policies in the cluster.
+
+## Storage Policy Satisfier Service
+
+- Support data migration across different storage policies.
+ - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
+- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
+- Responsible for managing storage policy-related migration work.
+- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+
+## Permissions Management
+
+- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+
+## Storage Policy Supported Replicas Types
+
+- Support replica type storage policies.
+- Support EC type storage policies.
+ - EC types cannot support some storage policies that mix different storage media.
+
+## Storage Policy Supported Bucket Types
+
+- Support all types of buckets.
+
+## Compatible HDFS Commands
+
+HDFS storage policy-related commands are not supported.
+
+Do not support specified storage policy when creating a file using HDFS Filesystem interface, the Hadoop FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy by Create method.
+
+- Ozone needs to record the access frequency of keys/buckets.
+
+## Recon Side Feature
+
+Support displaying relevant storage tier information in Recon. Need to design more detailed. // TODO
+
+# Architecture Design
+
+This section aims to describe the architecture and design concepts of the system without delving into specific implementation details. Its purpose is to validate the feasibility and rationality of the technical route, as well as to assess the workload and technical dependencies of related tasks.
+
+## Writing Key Process
+
+Writing data to the specified tier:
+
+- When a user creates a key, they set the storage policy and fallbackStrategy parameters and send the request to the OM:
+ - CreateKey
+ - CreateFile
+ - CreateMultipartKey
+- Community Ozone:
+ - CreateStreamKey
+ - CreateStreamFile
+ - CreateMultipartStreamKey
+- OM adds the storage policy information to the allocateBlock request sent to the SCM. The actual storage strategy should be calculated based on the storage strategy requirements above.
+- SCM, upon receiving the request, selects the Pipeline based on the storage policy and allocates a Container.
Review Comment:
Yes, client will send the StorageType to DN when write a key, a DN may contain both SSD and HDD storage devices, so the StorageType must be passed to the DN to enable creating/writing containers on the specified Volume type.
Additionally, the ContainerData for container replicas on the DN will include a storageType field. This allows DiskBalancer, ContainerBalancer, and ReplicationManager to determine the appropriate Volume type for migrating container replicas.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
Review Comment:
In our internal, the actual migration task will be executed by the Jobworker (A standalone service, just like S3G), configuring the number of worker threads for Jobworker can limit the migration speed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4109608481
@greenwich @chungen0126 @errose28 @vtutrinov The document has been updated. Please check it. All the content covered in the current document has been implemented in our internal version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3764801131
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
Review Comment:
DEEP_ARCHIVE is colder than GLACIER in AWS S3. Shall we map DEEP_ARCHIVE to Cold too here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3773018595
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
Review Comment:
> Container replica whose storage type mismatches the container's storage tier.
When will the Container replica's storage type mismatched with the container's storage tier?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4365281668
@xichen01 How about we cut a new HDDS-11233 branch? The current branch can be done without the transition patch ([HDDS-8342](https://issues.apache.org/jira/browse/HDDS-8342)). We can add the transition after HDDS-8342 is merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2098394743
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
Review Comment:
I meant org.apache.hadoop.hdds.scm.net.NodeSchema. Will the Storage Tier (aka `rack of specific storage volumes`) become a part of the network topology
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #6989: HDDS-11233. Ozone Storage Policy Support.
URL: https://github.com/apache/ozone/pull/6989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4351185647
@greenwich @xichen01 FYI, test in https://github.com/ivandika3/ozone/tree/refs/heads/backport-storage-policy-storage-class passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4353095024
> Regarding Phase 2, just thinking ahead — would it make sense to initially target a simpler mover-style implementation (similar in spirit to HDFS Mover) before introducing a separate job worker subsystem? That might allow basic Storage Policy Migration functionality to be delivered earlier and iterated on over time.
@greenwich Yes, we can first implement a standalone StoragePolicySatisfier (similar to HDFS Mover or HDFS StoragePolicySatisfier) without needing to implement a separate subsystem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4366105397
> @xichen01 How about we cut a new [HDDS-11233](https://issues.apache.org/jira/browse/HDDS-11233) branch? The current branch can be done without the transition patch ([HDDS-8342](https://issues.apache.org/jira/browse/HDDS-8342)). We can add the transition after [HDDS-8342](https://issues.apache.org/jira/browse/HDDS-8342) is merged to master.
I think we can create a new HDDS-11233 branch on the Apache Ozone repositories, just like https://github.com/apache/ozone/tree/HDDS-8342. Then we can merge related Commit to this branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4384175779
> @ivandika3 @greenwich @chungen0126 #10191 Please help to review
Looks good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-5281009761
@xichen01 , have we considered the future expand of storage policy, for example, to support HDFS like ONE_SSD policy, which has one replica on SSD, and other replicas on HDDS? If yes, what will be the change of the storage policy, and storage tier? If not, is it because this usage is not that popular in reality?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-3946550817
Hi @greenwich I see you have also opened #9807. If you would like to continue work on this, we should start by reaching agreement on a design doc. I'm not sure we finished that process yet. @ivandika3 @xichen01 does this doc need more updates/review? Should we continue work on it here or open a new PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-3953103047
@errose28 Thanks for noticing this PR.
@greenwich Thank you for the update.
### Regarding the current status of this PR
The current PR hasn't been updated for a while, mainly because there doesn't seem to be a strong demand for this feature from other members of the community, so reviews have stagnated.
However, this feature has been fully implemented internally, including support for StoragePolicy across all S3 and Filesystem write interfaces, as well as support for StoragePolicy in ReplicationManager and ContainerBalancer.
We've basically implemented it according to this design document (some parts of the document need updating; I can update the document if needed).
We also support S3 Lifecycle (https://issues.apache.org/jira/browse/HDDS-8342), allowing you to set a Lifecycle for a Bucket to migrate specified keys to a specified StoragePolicy at a specified time (including from SSD migrate to DISK, and also from THREE Replication migrate to EC) or similar HDFS SPS (Storage Policy Satisfaction) functionality.
### Follow-up
If you or others in the community are willing, we can continue to move forward with this PR, of course, you can also move forward with your own proposals, we can cooperate if you need it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-3947561978
@errose28 Thanks for looking! I am very sorry for the noise my PR caused; it wasn't intended to be public (My bad - I haven't set it to Draft when I created it).
Let me explain my motivation.
1. My team **needs** a storage policy and tiering support for Ozone. Unfortunately, reality is tough, and if we don't have it this year (ideally H1), then the Ozone might be deprioritised because we use other Ozone competitors in the company.
2. The patch attached to this pull request doesn't seem to be up to date with the master or even 2.0 or 2.1 releases.
3. As I can see, there has been no work on this PR since Nov 2025.
4. Plus, I might be wrong, but I got an impression that we want a perfect design and implementation here before we start coding.
5. My approach is different - I want to design and build it **incrementally** because that's the simplest way to adopt it in my team, start using it and collect the feedback from others. So
- I analysed the current state, reviewed the contents of the patch in this pull request, and created a small roadmap of the features I want in storage tiering.
- I prioritised them and split them into small releases. I called them MVP-1, MVP-2, etc. Each MVP should take me around 1 week to implement.
- Each MVP should have a complete set of features that work e2e. At the moment, I work on MVP-1.
- I planned to test MVPs on our PROD environment, and if it works as expected, then go back to the Ozone community and share what I have.
6. As I haven't created any Jira tickets because I thought of implementing and testing my MVP first. However, each independent feature is pushed as a separate commit, so it might later be retrofitted into different ASF jira tickeets (if needed).
7. The last thing, using Apache Ozone's PR feature is very useful for me because it triggers the whole CI and whatnot and allows seeing diff with the master to make my branch in sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-3985905946
@greenwich @chungen0126 @errose28 @vtutrinov I will update this document, adding more detailed content and incorporating some minor changes. I will try to complete it this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4436643699
> @ivandika3, @xichen01, as we have [HDDS-15239](https://issues.apache.org/jira/browse/HDDS-15239) merged and a new branch created. Are we ready to move forward?
Yes, we will continue to submit new PRs after the merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4295042380
@greenwich Sorry for the late reply. Let me try to use AI agent to backport this to my fork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3773258833
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
Review Comment:
RM normally has these situations,
Fallback
1. under replication yes
2. topology not satisfied ?
3. decommissioning yes
4. container balancer no
I think whether fallback is allowed depends on the case. Data integrity/safety has high priority over storage policy, so if data is under replication or decommissioning a datanode, fallback must be allowed. For container balancer, since it will evaluate utilization based on storage types, I guess it doesn't have fallback requirement. Topology is a question, topology helps to guarantee the data accessibility in case the hardware failure, in this case, it looks like it's priority is higher than storage policy too, so the answer seems yes too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-2988190589
@ivandika3 @xichen01 the diff above has references to the following non-existent files (relative to ozone-1.4.1):
```
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementDataCenterAware.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/SCMContainerPlacementDataRecovery.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelinePlacementDataCenterAware.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementDataCenterAware.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementDataCenterAwareSpecialCase.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementDataRecovery.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementDcFlow.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestPipelinePlacementDataCenterAware.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/diskbalancer/TestDiskBalancerService.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/placement/algorithms/TestSCMContainerPlacementStorageTier.java
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/StorageTierUtil.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeUtils.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/TestSpecialCloseContainerEventHandler.java
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/TestPeriodicContainerCloser.java
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/AbstractRootedOzoneFileSystemTest.java
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/UpdateBucketOptions.java
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/StorageTypeUtils.java
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneStoragePolicy.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/TestContainerBalancerTaskDcFlow.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerUtils.java
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/client/StorageTierUtilTest.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/dcflow/ContainerBalancerSelectionCriteriaDcFlow.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/AbstractStorageTypeChoosingPolicy.java
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/storagepolicy/StoragePolicyCommands.java
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/storagepolicy/UsageInfoSubCommand.java
```
Could you provide them too, or point me to the commit where I can fetch them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-2955463014
@vtutrinov the fastest way I think we can do is to provide you with the diffs. However this diff won't apply cleanly on the master branch since our branch is based on 1.4.1 version with some of our internal specific changes. I probably can provide some of it this weekend.
Feature branch in my fork might take a while since we need to resolve the conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-2976630111
@ivandika3 I don't want to rush, but is there any news about the mentioned diff?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2097266132
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
Review Comment:
Will we deal with the storage tier as an entry of the cluster topology?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
Review Comment:
Is this the final list of desired/planned storage policies? Wouldn't we like to implement policies like in HDFS - https://hadoop.apache.org/docs/r3.4.1/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html ?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
+ - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
+ - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
+ - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
+- `ozone admin container info`, supports displaying the storage policy of Containers.
+- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
+- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
+- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
+- Datanode volume related Metrics support displaying the storage policies of volumes.
+- SCM supports displaying the data space information of different storage policies in the cluster.
+
+## Storage Policy Satisfier Service
+
+- Support data migration across different storage policies.
+ - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
+- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
+- Responsible for managing storage policy-related migration work.
+- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+
+## Permissions Management
+
+- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+
+## Storage Policy Supported Replicas Types
+
+- Support replica type storage policies.
+- Support EC type storage policies.
+ - EC types cannot support some storage policies that mix different storage media.
+
+## Storage Policy Supported Bucket Types
+
+- Support all types of buckets.
+
+## Compatible HDFS Commands
+
+HDFS storage policy-related commands are not supported.
+
+Do not support specified storage policy when creating a file using HDFS Filesystem interface, the Hadoop FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy by Create method.
+
+- Ozone needs to record the access frequency of keys/buckets.
+
+## Recon Side Feature
+
+Support displaying relevant storage tier information in Recon. Need to design more detailed. // TODO
+
+# Architecture Design
+
+This section aims to describe the architecture and design concepts of the system without delving into specific implementation details. Its purpose is to validate the feasibility and rationality of the technical route, as well as to assess the workload and technical dependencies of related tasks.
+
+## Writing Key Process
+
+Writing data to the specified tier:
+
+- When a user creates a key, they set the storage policy and fallbackStrategy parameters and send the request to the OM:
+ - CreateKey
+ - CreateFile
+ - CreateMultipartKey
+- Community Ozone:
+ - CreateStreamKey
+ - CreateStreamFile
+ - CreateMultipartStreamKey
+- OM adds the storage policy information to the allocateBlock request sent to the SCM. The actual storage strategy should be calculated based on the storage strategy requirements above.
+- SCM, upon receiving the request, selects the Pipeline based on the storage policy and allocates a Container.
+ - The storage type of each Container replicas should be added to for the Datanodes in the Pipeline, just like the replicaIndex for the EC key.
+ - If the expected storage tier Pipeline cannot be found, attempt to find the fallback tier defined by the storage policy if the fallbackStrategy is allowed. If it cannot be found, the creation process fails.
+ - If the SCM needs to create a new Container for a specific storage tier, SCM needs to record the Container’s storage tier for the Container. So the ContainerData should add a storage tier field.
+- After receiving the Pipeline with the specified storage type information, the client writes the Chunk replica with the storage type information.
+- Datanode writes data to the specified Container replica or creates a Container replica in the specified volume based on the storage type .
+- OMKeyInfo needs to persist in the storage policy field, if the storage policy does not be specified, the storage policy should be null.
+
+## DN Container Replica Creation
+
+- DN should select the appropriate volume based on the storage type.
+- If the Container includes storage type information, DN should store this information in the Container's replicas YAML configuration.
+- When DN starts, it should read the storage type information from the Container's YAML configuration and load it into memory.
+- If the storage type of Container is not specified, it defaults to Disk.
+
+## Storage Policy Management
+
+- Implement key, prefix, and bucket storage policy-related commands as needed, and add related fields to persist tier information.
+
+## SCM Pipeline / ContainerReplica Management and Block Allocation
+
+- Pipeline needs to add tier-related fields to distinguish between different tiers of Pipelines.
+- For Pipelines tier:
Review Comment:
The current implementation of the background pipeline creator creates a limited list of RATIS/THREE-pipelines per datanode. The design doc proposes to deal with specific pipelines for different storage policies. Is there any proposal doc on how we gonna deal with the limitation mentioned earlier? (How should we deal with the distribution of the pipelines of different types: 30% per storage tire or some other option? What if the pipelines of one of the storrage tire will not be used?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2873208891
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
Review Comment:
Currently we only implement few storage policy types, but it is easy to add new storage policy in the code. we Do not support custom storage policy by user, but this implementation should not be difficult.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
Review Comment:
Yes, support for FSO and OBS will be added. This part needs to be updated.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
Review Comment:
For the key migration, our internal implementation utilizes S3 Lifecycle rules by configuring a rule for the bucket. OzoneManager's LifecycleService checks whether keys comply with the storage policy based on this rule. If non-compliant, it generates a command to be executed by Jobworker for migration. Upon receiving the command, Jobworker performs the migration via the `rewriteKey` API.
If Jobworker is not used, a local client must be implemented to inspect and migrate keys via the `rewriteKey` API.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
+ - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
+ - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
+ - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
+- `ozone admin container info`, supports displaying the storage policy of Containers.
+- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
+- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
+- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
+- Datanode volume related Metrics support displaying the storage policies of volumes.
+- SCM supports displaying the data space information of different storage policies in the cluster.
+
+## Storage Policy Satisfier Service
+
+- Support data migration across different storage policies.
+ - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
+- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
+- Responsible for managing storage policy-related migration work.
+- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+
+## Permissions Management
+
+- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+
+## Storage Policy Supported Replicas Types
+
+- Support replica type storage policies.
+- Support EC type storage policies.
+ - EC types cannot support some storage policies that mix different storage media.
+
+## Storage Policy Supported Bucket Types
+
+- Support all types of buckets.
+
+## Compatible HDFS Commands
+
+HDFS storage policy-related commands are not supported.
+
+Do not support specified storage policy when creating a file using HDFS Filesystem interface, the Hadoop FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy by Create method.
+
+- Ozone needs to record the access frequency of keys/buckets.
+
+## Recon Side Feature
+
+Support displaying relevant storage tier information in Recon. Need to design more detailed. // TODO
+
+# Architecture Design
+
+This section aims to describe the architecture and design concepts of the system without delving into specific implementation details. Its purpose is to validate the feasibility and rationality of the technical route, as well as to assess the workload and technical dependencies of related tasks.
+
+## Writing Key Process
+
+Writing data to the specified tier:
+
+- When a user creates a key, they set the storage policy and fallbackStrategy parameters and send the request to the OM:
+ - CreateKey
+ - CreateFile
+ - CreateMultipartKey
+- Community Ozone:
+ - CreateStreamKey
+ - CreateStreamFile
+ - CreateMultipartStreamKey
+- OM adds the storage policy information to the allocateBlock request sent to the SCM. The actual storage strategy should be calculated based on the storage strategy requirements above.
+- SCM, upon receiving the request, selects the Pipeline based on the storage policy and allocates a Container.
+ - The storage type of each Container replicas should be added to for the Datanodes in the Pipeline, just like the replicaIndex for the EC key.
+ - If the expected storage tier Pipeline cannot be found, attempt to find the fallback tier defined by the storage policy if the fallbackStrategy is allowed. If it cannot be found, the creation process fails.
+ - If the SCM needs to create a new Container for a specific storage tier, SCM needs to record the Container’s storage tier for the Container. So the ContainerData should add a storage tier field.
+- After receiving the Pipeline with the specified storage type information, the client writes the Chunk replica with the storage type information.
+- Datanode writes data to the specified Container replica or creates a Container replica in the specified volume based on the storage type .
+- OMKeyInfo needs to persist in the storage policy field, if the storage policy does not be specified, the storage policy should be null.
+
+## DN Container Replica Creation
+
+- DN should select the appropriate volume based on the storage type.
+- If the Container includes storage type information, DN should store this information in the Container's replicas YAML configuration.
+- When DN starts, it should read the storage type information from the Container's YAML configuration and load it into memory.
+- If the storage type of Container is not specified, it defaults to Disk.
+
+## Storage Policy Management
+
+- Implement key, prefix, and bucket storage policy-related commands as needed, and add related fields to persist tier information.
+
+## SCM Pipeline / ContainerReplica Management and Block Allocation
+
+- Pipeline needs to add tier-related fields to distinguish between different tiers of Pipelines.
+- For Pipelines tier:
+ - Pipeline adds tier attribute, Pipeline belongs to a specific tier
+ - If the Pipeline tier is null, then the Pipeline tier is Disk, so all the Pipeline tier created before storage policy release should be Disk.
+ - After the storage policy is released, the Container which is allocated by the existing Pipeline may not be created normally due to the change of the storage type of the Datanode volume. For example, if all volumes on a Datanode of the existing Pipeline1 are configured as SSD, but the tier of the existing Pipeline1 is Disk and the storage type of the Container replicas is Disk, the creation of this Container will fail because no suitable volume can be found.
+ - For the above scenario the Pipeline should be closed and SCM should create a new Pipeline.
+ - SCM should select the appropriate Pipeline based on the storage policy selection information in the request during getPipelines.
+ - If no suitable Pipeline is found, select the appropriate Datanode to create the Pipeline.
+ - If no suitable Datanode is found, fallback to creating a Pipeline for the defined fallback tier Pipeline.
+ - If a fallback Pipeline tier cannot be created, AllocateBlock fails.
+ - Currently, Pipelines are selected using a polling method (PipelineStateMap#getPipelines). If there are many Pipelines, performance may be affected, so a more efficient method can be considered.
+
+## Datanode Heartbeat Reporting Process
+
+- Datanode reports the storage type information of each volume through heartbeats. SCM aggregates and manages Datanodes based on different storage types, including the remaining capacity of each storage type.
+ - Currently, Datanode reports volume capacity information through NodeReport. The storage type information of volume can be reported to SCM in DATA_VOLUME of getNodeReport.
Review Comment:
Currently, there is no quota mechanism similar to bucket quotas that physically limits usage per tier.
However, you can use `ozone admin datanode list` to view the specific usage details for each tier.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
+ - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
+ - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
+ - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
+- `ozone admin container info`, supports displaying the storage policy of Containers.
+- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
+- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
+- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
+- Datanode volume related Metrics support displaying the storage policies of volumes.
+- SCM supports displaying the data space information of different storage policies in the cluster.
+
+## Storage Policy Satisfier Service
+
+- Support data migration across different storage policies.
+ - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
+- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
+- Responsible for managing storage policy-related migration work.
+- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+
+## Permissions Management
+
+- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+
+## Storage Policy Supported Replicas Types
+
+- Support replica type storage policies.
+- Support EC type storage policies.
+ - EC types cannot support some storage policies that mix different storage media.
Review Comment:
Introducing a Storage Policy does not alter the pipeline's behavioral logic; it only affects the nodes composing the pipeline.
The pipeline will add a Storage Tier attribute. For a pipeline with an SSD Storage Tier attribute, it will be composed of DNs that support SSD.
The primary logical change occurs during pipeline creation, where DNs with suitable volumes are selected to form the pipeline. Read/write behavior after pipeline creation remains unchanged.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
+ - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
+ - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
+ - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
+- `ozone admin container info`, supports displaying the storage policy of Containers.
+- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
+- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
+- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
+- Datanode volume related Metrics support displaying the storage policies of volumes.
+- SCM supports displaying the data space information of different storage policies in the cluster.
+
+## Storage Policy Satisfier Service
+
+- Support data migration across different storage policies.
+ - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
+- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
+- Responsible for managing storage policy-related migration work.
+- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+
+## Permissions Management
+
+- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+
+## Storage Policy Supported Replicas Types
+
+- Support replica type storage policies.
+- Support EC type storage policies.
+ - EC types cannot support some storage policies that mix different storage media.
+
+## Storage Policy Supported Bucket Types
+
+- Support all types of buckets.
+
+## Compatible HDFS Commands
+
+HDFS storage policy-related commands are not supported.
+
+Do not support specified storage policy when creating a file using HDFS Filesystem interface, the Hadoop FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy by Create method.
+
+- Ozone needs to record the access frequency of keys/buckets.
+
+## Recon Side Feature
+
+Support displaying relevant storage tier information in Recon. Need to design more detailed. // TODO
+
+# Architecture Design
+
+This section aims to describe the architecture and design concepts of the system without delving into specific implementation details. Its purpose is to validate the feasibility and rationality of the technical route, as well as to assess the workload and technical dependencies of related tasks.
+
+## Writing Key Process
+
+Writing data to the specified tier:
+
+- When a user creates a key, they set the storage policy and fallbackStrategy parameters and send the request to the OM:
+ - CreateKey
+ - CreateFile
+ - CreateMultipartKey
+- Community Ozone:
+ - CreateStreamKey
+ - CreateStreamFile
+ - CreateMultipartStreamKey
+- OM adds the storage policy information to the allocateBlock request sent to the SCM. The actual storage strategy should be calculated based on the storage strategy requirements above.
+- SCM, upon receiving the request, selects the Pipeline based on the storage policy and allocates a Container.
+ - The storage type of each Container replicas should be added to for the Datanodes in the Pipeline, just like the replicaIndex for the EC key.
+ - If the expected storage tier Pipeline cannot be found, attempt to find the fallback tier defined by the storage policy if the fallbackStrategy is allowed. If it cannot be found, the creation process fails.
+ - If the SCM needs to create a new Container for a specific storage tier, SCM needs to record the Container’s storage tier for the Container. So the ContainerData should add a storage tier field.
+- After receiving the Pipeline with the specified storage type information, the client writes the Chunk replica with the storage type information.
+- Datanode writes data to the specified Container replica or creates a Container replica in the specified volume based on the storage type .
+- OMKeyInfo needs to persist in the storage policy field, if the storage policy does not be specified, the storage policy should be null.
+
+## DN Container Replica Creation
+
+- DN should select the appropriate volume based on the storage type.
+- If the Container includes storage type information, DN should store this information in the Container's replicas YAML configuration.
+- When DN starts, it should read the storage type information from the Container's YAML configuration and load it into memory.
+- If the storage type of Container is not specified, it defaults to Disk.
+
+## Storage Policy Management
+
+- Implement key, prefix, and bucket storage policy-related commands as needed, and add related fields to persist tier information.
+
+## SCM Pipeline / ContainerReplica Management and Block Allocation
+
+- Pipeline needs to add tier-related fields to distinguish between different tiers of Pipelines.
+- For Pipelines tier:
Review Comment:
Ratis' Pipeline utilizes a Background service for creation (`BackgroundPipelineCreator`). The number of Ratis Pipelines can be controlled through configuration.
EC-type Pipelines are created only when used. Different EC configurations generate distinct Pipelines. Pipelines from different EC configurations do not interfere with each other. (`WritableECContainerProvider#getContainer`) SCM selects the DN with the appropriate Storage Type to create the Pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2875959913
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
+ - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
+ - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
+ - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
+- `ozone admin container info`, supports displaying the storage policy of Containers.
+- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
+- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
+- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
+- Datanode volume related Metrics support displaying the storage policies of volumes.
+- SCM supports displaying the data space information of different storage policies in the cluster.
+
+## Storage Policy Satisfier Service
+
+- Support data migration across different storage policies.
+ - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
+- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
+- Responsible for managing storage policy-related migration work.
+- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+
+## Permissions Management
+
+- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+
+## Storage Policy Supported Replicas Types
+
+- Support replica type storage policies.
+- Support EC type storage policies.
+ - EC types cannot support some storage policies that mix different storage media.
Review Comment:
Understand: am I right that in that case, datanodes with heterogeneous volume distributions are preferred (DNs that have both SSDs and DISKs)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4319936554
Thanks @greenwich , @xichen01 would you mind take a look at that?
I'm currently backporting it in my fork, but the test are still failing (https://github.com/ivandika3/ozone/tree/refs/heads/backport-storage-policy-storage-class)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3772977410
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
Review Comment:
is any state container allowed here to set the storage tier?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-2915981608
@xichen01 @kerneltime @sodonnel, could you help somehow to force the review of the design doc? The feature is very needed, and I would gladly start implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-2953554337
@vtutrinov Currently the implementation is being worked on internally for the past one year.
The basic implementation of storage policy and storage types integration on containers, pipelines, volumes, s3 storage class, and creating key / file with storage policy has been implemented but still need extensive testing. Currently we are focusing on storage policy migration implementation.
@xichen01 would know more about the approximate timestamps, but we hope to have a working implementation in the next quarter (i.e. Q3 2025) or so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4317800028
> @greenwich Sorry for the late reply. Let me try to use an AI agent to backport the storage policy to my fork.
@ivandika3, I managed to do a similar thing because I wanted to prepare everything up front to make things go faster for this PR.
- I picked the patch attached to this PR (that was branched out from ozone 1.4)
- Merged it to sync with master around two weeks ago, fixed all the conflicts and broken tests
- Yesterday I synced upstream's master with that branch again to be up to date.
- Please have a look, maybe it already has what you were planning to do: https://github.com/greenwich/ozone/tree/refs/heads/HDDS-11233_patch_merged
- All tests are green: https://github.com/greenwich/ozone/actions/runs/24915662451
cc: @xichen01
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-3956283360
> the diff above has references to the following non-existent files (relative to ozone-1.4.1):
These refer to our internal placement policy to support multi-DC setup and can be ignored as these do not pertain the any current functionality in community Ozone. The patch diff serve only as an overview of what the changes might look like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-3956982569
@ivandika3 Thanks for the clarification — that makes sense.
I fully understand that contributions are voluntary, and I really appreciate the time and effort everyone is putting into this.
I’m very interested in helping move this forward and would be glad to contribute where it makes the most sense.
Regarding Phase 2, just thinking ahead — would it make sense to initially target a simpler mover-style implementation (similar in spirit to HDFS Mover) before introducing a separate job worker subsystem? That might allow basic Storage Policy Migration functionality to be delivered earlier and iterated on over time.
Of course, I’m happy to align with the broader design direction — just exploring whether an incremental path could also work here.
Please let me know how I can best contribute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4364194566
> @greenwich @xichen01 FYI, test in https://github.com/ivandika3/ozone/tree/refs/heads/backport-storage-policy-storage-class passed. You can refer to the diff in https://github.com/ivandika3/ozone/pull/4
@ivandika3 We can create corresponding sub tasks for these commits and then merge them into the HDDS-8342 branch via MR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2870767609
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
Review Comment:
Is there a plan to add more storage tiers beyond SSDs, such as NVME? So then it becomes Hot, and SSDs are Warm? And/Or give a user the ability to create their own storage policies and name their own tiers?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
Review Comment:
Correct me if I am wrong, but the current patch has the following limitation:
If an operator changes a bucket's policy from WARM to HOT, and teams decide to create their own mover or policy-satisfier service.
1. There's no command/API to trigger bulk migration
2. There's no way to discover which keys are on the wrong tier (except checking one key at a time)
3. The only migration path is writing a custom script that calls RewriteKey per key
4. There is no container-level migration, too?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
Review Comment:
Document doesn't mention OSB buckets, but I assume they are also supported, right?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
+ - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
+ - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
+ - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
+- `ozone admin container info`, supports displaying the storage policy of Containers.
+- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
+- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
+- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
+- Datanode volume related Metrics support displaying the storage policies of volumes.
+- SCM supports displaying the data space information of different storage policies in the cluster.
+
+## Storage Policy Satisfier Service
+
+- Support data migration across different storage policies.
+ - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
+- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
+- Responsible for managing storage policy-related migration work.
+- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+
+## Permissions Management
+
+- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+
+## Storage Policy Supported Replicas Types
+
+- Support replica type storage policies.
+- Support EC type storage policies.
+ - EC types cannot support some storage policies that mix different storage media.
+
+## Storage Policy Supported Bucket Types
+
+- Support all types of buckets.
+
+## Compatible HDFS Commands
+
+HDFS storage policy-related commands are not supported.
+
+Do not support specified storage policy when creating a file using HDFS Filesystem interface, the Hadoop FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy by Create method.
+
+- Ozone needs to record the access frequency of keys/buckets.
+
+## Recon Side Feature
+
+Support displaying relevant storage tier information in Recon. Need to design more detailed. // TODO
+
+# Architecture Design
+
+This section aims to describe the architecture and design concepts of the system without delving into specific implementation details. Its purpose is to validate the feasibility and rationality of the technical route, as well as to assess the workload and technical dependencies of related tasks.
+
+## Writing Key Process
+
+Writing data to the specified tier:
+
+- When a user creates a key, they set the storage policy and fallbackStrategy parameters and send the request to the OM:
+ - CreateKey
+ - CreateFile
+ - CreateMultipartKey
+- Community Ozone:
+ - CreateStreamKey
+ - CreateStreamFile
+ - CreateMultipartStreamKey
+- OM adds the storage policy information to the allocateBlock request sent to the SCM. The actual storage strategy should be calculated based on the storage strategy requirements above.
+- SCM, upon receiving the request, selects the Pipeline based on the storage policy and allocates a Container.
+ - The storage type of each Container replicas should be added to for the Datanodes in the Pipeline, just like the replicaIndex for the EC key.
+ - If the expected storage tier Pipeline cannot be found, attempt to find the fallback tier defined by the storage policy if the fallbackStrategy is allowed. If it cannot be found, the creation process fails.
+ - If the SCM needs to create a new Container for a specific storage tier, SCM needs to record the Container’s storage tier for the Container. So the ContainerData should add a storage tier field.
+- After receiving the Pipeline with the specified storage type information, the client writes the Chunk replica with the storage type information.
+- Datanode writes data to the specified Container replica or creates a Container replica in the specified volume based on the storage type .
+- OMKeyInfo needs to persist in the storage policy field, if the storage policy does not be specified, the storage policy should be null.
+
+## DN Container Replica Creation
+
+- DN should select the appropriate volume based on the storage type.
+- If the Container includes storage type information, DN should store this information in the Container's replicas YAML configuration.
+- When DN starts, it should read the storage type information from the Container's YAML configuration and load it into memory.
+- If the storage type of Container is not specified, it defaults to Disk.
+
+## Storage Policy Management
+
+- Implement key, prefix, and bucket storage policy-related commands as needed, and add related fields to persist tier information.
+
+## SCM Pipeline / ContainerReplica Management and Block Allocation
+
+- Pipeline needs to add tier-related fields to distinguish between different tiers of Pipelines.
+- For Pipelines tier:
Review Comment:
However, on a dedicated-tier cluster (16 SSD, 16 DISK nodes), the SSD tier can only form a limited number of EC 3+2 pipelines. Does the background creator have a target count? Does it back off when the tier is saturated?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
+ - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
+ - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
+ - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
+- `ozone admin container info`, supports displaying the storage policy of Containers.
+- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
+- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
+- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
+- Datanode volume related Metrics support displaying the storage policies of volumes.
+- SCM supports displaying the data space information of different storage policies in the cluster.
+
+## Storage Policy Satisfier Service
+
+- Support data migration across different storage policies.
+ - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
+- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
+- Responsible for managing storage policy-related migration work.
+- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+
+## Permissions Management
+
+- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+
+## Storage Policy Supported Replicas Types
+
+- Support replica type storage policies.
+- Support EC type storage policies.
+ - EC types cannot support some storage policies that mix different storage media.
Review Comment:
- What is the minimum node count per tier for each replication config? (EC 3+2 needs at least 5 nodes, ideally across 3+ racks)
- Should SCM refuse to enable a tier if the node count is below the minimum?
- How does pipeline closure cascade when a tier loses nodes — does it close all pipelines on that tier or just affected ones?
- What happens to in-flight writes when the last pipeline on a tier closes — does fallback kick in immediately, or does the write fail?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
+ - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
+ - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
+ - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
+- `ozone admin container info`, supports displaying the storage policy of Containers.
+- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
+- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
+- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
+- Datanode volume related Metrics support displaying the storage policies of volumes.
+- SCM supports displaying the data space information of different storage policies in the cluster.
+
+## Storage Policy Satisfier Service
+
+- Support data migration across different storage policies.
+ - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
+- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
+- Responsible for managing storage policy-related migration work.
+- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+
+## Permissions Management
+
+- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+
+## Storage Policy Supported Replicas Types
+
+- Support replica type storage policies.
+- Support EC type storage policies.
+ - EC types cannot support some storage policies that mix different storage media.
+
+## Storage Policy Supported Bucket Types
+
+- Support all types of buckets.
+
+## Compatible HDFS Commands
+
+HDFS storage policy-related commands are not supported.
+
+Do not support specified storage policy when creating a file using HDFS Filesystem interface, the Hadoop FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy by Create method.
+
+- Ozone needs to record the access frequency of keys/buckets.
+
+## Recon Side Feature
+
+Support displaying relevant storage tier information in Recon. Need to design more detailed. // TODO
+
+# Architecture Design
+
+This section aims to describe the architecture and design concepts of the system without delving into specific implementation details. Its purpose is to validate the feasibility and rationality of the technical route, as well as to assess the workload and technical dependencies of related tasks.
+
+## Writing Key Process
+
+Writing data to the specified tier:
+
+- When a user creates a key, they set the storage policy and fallbackStrategy parameters and send the request to the OM:
+ - CreateKey
+ - CreateFile
+ - CreateMultipartKey
+- Community Ozone:
+ - CreateStreamKey
+ - CreateStreamFile
+ - CreateMultipartStreamKey
+- OM adds the storage policy information to the allocateBlock request sent to the SCM. The actual storage strategy should be calculated based on the storage strategy requirements above.
+- SCM, upon receiving the request, selects the Pipeline based on the storage policy and allocates a Container.
+ - The storage type of each Container replicas should be added to for the Datanodes in the Pipeline, just like the replicaIndex for the EC key.
+ - If the expected storage tier Pipeline cannot be found, attempt to find the fallback tier defined by the storage policy if the fallbackStrategy is allowed. If it cannot be found, the creation process fails.
+ - If the SCM needs to create a new Container for a specific storage tier, SCM needs to record the Container’s storage tier for the Container. So the ContainerData should add a storage tier field.
+- After receiving the Pipeline with the specified storage type information, the client writes the Chunk replica with the storage type information.
+- Datanode writes data to the specified Container replica or creates a Container replica in the specified volume based on the storage type .
+- OMKeyInfo needs to persist in the storage policy field, if the storage policy does not be specified, the storage policy should be null.
+
+## DN Container Replica Creation
+
+- DN should select the appropriate volume based on the storage type.
+- If the Container includes storage type information, DN should store this information in the Container's replicas YAML configuration.
+- When DN starts, it should read the storage type information from the Container's YAML configuration and load it into memory.
+- If the storage type of Container is not specified, it defaults to Disk.
+
+## Storage Policy Management
+
+- Implement key, prefix, and bucket storage policy-related commands as needed, and add related fields to persist tier information.
+
+## SCM Pipeline / ContainerReplica Management and Block Allocation
+
+- Pipeline needs to add tier-related fields to distinguish between different tiers of Pipelines.
+- For Pipelines tier:
+ - Pipeline adds tier attribute, Pipeline belongs to a specific tier
+ - If the Pipeline tier is null, then the Pipeline tier is Disk, so all the Pipeline tier created before storage policy release should be Disk.
+ - After the storage policy is released, the Container which is allocated by the existing Pipeline may not be created normally due to the change of the storage type of the Datanode volume. For example, if all volumes on a Datanode of the existing Pipeline1 are configured as SSD, but the tier of the existing Pipeline1 is Disk and the storage type of the Container replicas is Disk, the creation of this Container will fail because no suitable volume can be found.
+ - For the above scenario the Pipeline should be closed and SCM should create a new Pipeline.
+ - SCM should select the appropriate Pipeline based on the storage policy selection information in the request during getPipelines.
+ - If no suitable Pipeline is found, select the appropriate Datanode to create the Pipeline.
+ - If no suitable Datanode is found, fallback to creating a Pipeline for the defined fallback tier Pipeline.
+ - If a fallback Pipeline tier cannot be created, AllocateBlock fails.
+ - Currently, Pipelines are selected using a polling method (PipelineStateMap#getPipelines). If there are many Pipelines, performance may be affected, so a more efficient method can be considered.
+
+## Datanode Heartbeat Reporting Process
+
+- Datanode reports the storage type information of each volume through heartbeats. SCM aggregates and manages Datanodes based on different storage types, including the remaining capacity of each storage type.
+ - Currently, Datanode reports volume capacity information through NodeReport. The storage type information of volume can be reported to SCM in DATA_VOLUME of getNodeReport.
Review Comment:
Will we have a quota or capacity awareness per tier?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-3549968542
Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2098242333
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
Review Comment:
Yes, we only support these three simple storage policies current, we can extend it in the future.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
Review Comment:
What does “deal with as a cluster topology” mean?
Storage Tier is the property of Pipeline and Container, when we create the key we will select the matched Storage Tier Pipeline and Container based on the key Storage Policy.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
+ - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
+ - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
+ - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
+- `ozone admin container info`, supports displaying the storage policy of Containers.
+- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
+- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
+- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
+- Datanode volume related Metrics support displaying the storage policies of volumes.
+- SCM supports displaying the data space information of different storage policies in the cluster.
+
+## Storage Policy Satisfier Service
+
+- Support data migration across different storage policies.
+ - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
+- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
+- Responsible for managing storage policy-related migration work.
+- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+
+## Permissions Management
+
+- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+
+## Storage Policy Supported Replicas Types
+
+- Support replica type storage policies.
+- Support EC type storage policies.
+ - EC types cannot support some storage policies that mix different storage media.
+
+## Storage Policy Supported Bucket Types
+
+- Support all types of buckets.
+
+## Compatible HDFS Commands
+
+HDFS storage policy-related commands are not supported.
+
+Do not support specified storage policy when creating a file using HDFS Filesystem interface, the Hadoop FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy by Create method.
+
+- Ozone needs to record the access frequency of keys/buckets.
+
+## Recon Side Feature
+
+Support displaying relevant storage tier information in Recon. Need to design more detailed. // TODO
+
+# Architecture Design
+
+This section aims to describe the architecture and design concepts of the system without delving into specific implementation details. Its purpose is to validate the feasibility and rationality of the technical route, as well as to assess the workload and technical dependencies of related tasks.
+
+## Writing Key Process
+
+Writing data to the specified tier:
+
+- When a user creates a key, they set the storage policy and fallbackStrategy parameters and send the request to the OM:
+ - CreateKey
+ - CreateFile
+ - CreateMultipartKey
+- Community Ozone:
+ - CreateStreamKey
+ - CreateStreamFile
+ - CreateMultipartStreamKey
+- OM adds the storage policy information to the allocateBlock request sent to the SCM. The actual storage strategy should be calculated based on the storage strategy requirements above.
+- SCM, upon receiving the request, selects the Pipeline based on the storage policy and allocates a Container.
+ - The storage type of each Container replicas should be added to for the Datanodes in the Pipeline, just like the replicaIndex for the EC key.
+ - If the expected storage tier Pipeline cannot be found, attempt to find the fallback tier defined by the storage policy if the fallbackStrategy is allowed. If it cannot be found, the creation process fails.
+ - If the SCM needs to create a new Container for a specific storage tier, SCM needs to record the Container’s storage tier for the Container. So the ContainerData should add a storage tier field.
+- After receiving the Pipeline with the specified storage type information, the client writes the Chunk replica with the storage type information.
+- Datanode writes data to the specified Container replica or creates a Container replica in the specified volume based on the storage type .
+- OMKeyInfo needs to persist in the storage policy field, if the storage policy does not be specified, the storage policy should be null.
+
+## DN Container Replica Creation
+
+- DN should select the appropriate volume based on the storage type.
+- If the Container includes storage type information, DN should store this information in the Container's replicas YAML configuration.
+- When DN starts, it should read the storage type information from the Container's YAML configuration and load it into memory.
+- If the storage type of Container is not specified, it defaults to Disk.
+
+## Storage Policy Management
+
+- Implement key, prefix, and bucket storage policy-related commands as needed, and add related fields to persist tier information.
+
+## SCM Pipeline / ContainerReplica Management and Block Allocation
+
+- Pipeline needs to add tier-related fields to distinguish between different tiers of Pipelines.
+- For Pipelines tier:
Review Comment:
The count limit for current Pipeline count will be extend to each storage tier, so in the extreme condition, there will be three times Pipeline in a cluster.
But in the actual situation, there may not be so many Pipelines, because a Pipeline may support multiple tiers, such as DISK and SSD, because there may be both SSD and DISK type Volume in a Datanode machine.
And the background Pipeline creator will check the current cluster's Volume type, if a cluster has only SSD Volume, the background Pipeline will create a Pipeline with only SSD Tier type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-3956078125
@xichen01 Let's reopen this patch and move this forward. I am willing spend some time on the storage policy support development. Hopefully other community members (@errose28 @chungen0126) are also able to help to push the review process.
@greenwich As mentioned in the https://github.com/apache/ozone/discussions/8811#discussioncomment-13928693 discussion last year. There are mainly two phases of Ozone Storage Policy Support
1. Supporting storage policy and storage types on Ozone
2. Storage Policy Migration support
Phase 1 goal is to allow client to upload keys with different storage policy and integrate storage policy and types to all Ozone pipeline, containers, etc. This will be our short term focus now since we are not introducing any new subsystem
Phase 2 goal is to support Storage Policy Migration. However, our current implementation requires a separate job worker subsystem. This will be longer term since we are introducing some new subsystem. If you don't require a separate job worker subsystem, you might need to write your own implementation of "Storage Policy Satisfier".
That said, hope you understand that all contributions to Ozone are purely voluntary and made by members with other higher level priorities and therefore we cannot 100% guarantee that this will be done in a timely manner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4378698452
> @xichen01 Could you help to take over? I'm currently quite busy and my GH account was recently blocked from GH actions (still waiting for support) so no CI can be triggered.
OK, I will
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2982425450
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
Review Comment:
This hasn't been implemented yet, but we can implement it if needed.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
Review Comment:
The prefix has not yet been implemented. If the prefix is implemented, the priority order will be `key → prefix → bucket → config`.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
Review Comment:
This point was omitted here; in fact, SCM also uses storage tiers to filter containers.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
Review Comment:
A new command (`SetContainerStorageTypeCommand`): When a Datanode receives this command, it updates the container replica, including both the in-memory data and the YAML file.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
+ Datanode, it will try to use the fallback storage types defined by the SCM container's storage tier. If the storage
+ tier has no defined fallback storage types, or no Datanodes matching the fallback storage types can be found, the
+ operation fails.
+
+## DiskBalancerService
+
+- DiskBalancerService calculates space usage and migrates containers between volumes of the same storage type within a
+ Datanode.
+
+# Configurations
+
+New configuration keys introduced by the storage policy feature:
+
+- `ozone.default.storagepolicy`
+ Default value: WARM. The default storage policy for writing keys when neither the key nor the bucket has a storage
+ policy set.
+
+- `ozone.default.storageTier`
+ Default value: DISK. The default storage tier.
+
+- `ozone.s3.default.storagepolicy`
+ Default value: STANDARD_IA. The default S3 storage class when StorageClass is not specified in the S3 request and
+ the bucket does not have a storage policy set. STANDARD_IA maps to Ozone storage policy WARM.
+
+- `hdds.datanode.default.storagetype`
+ Default value: DISK. The default storage type used when creating a container replica on a Datanode without a
+ specified storage type. This typically applies to write requests from older clients that do not support the storage
+ policy feature.
+
+- `ozone.scm.container.allow.null.storage.tier`
+ Default value: false. Whether to allow selecting containers with a null storage tier during container allocation.
+ This can be used during the upgrade period to allow new keys to be written to existing containers that do not yet
+ have a storage tier set.
+
+For a pure SSD cluster (where all Datanode volumes are configured with SSD storage type), the following configurations
+should be updated accordingly:
+
+- `ozone.default.storagepolicy` should be set to HOT (Hot maps to SSD storage tier).
+- `ozone.default.storageTier` should be set to SSD.
+- `ozone.s3.default.storagepolicy` should be set to STANDARD (STANDARD maps to Hot storage policy).
+- `hdds.datanode.default.storagetype` should be set to SSD.
+
+Otherwise, when writing a key without specifying a storage policy, the system will default to the WARM storage policy,
+which corresponds to the DISK storage tier. Since WARM does not define a fallback tier (its fallback is EMPTY), SCM
+cannot fall back to another tier, and the write will fail because no DISK type volumes are available in the cluster.
+
+Note that this situation only occurs when the volumes are explicitly configured as SSD in the `hdds.datanode.dir`
+configuration (e.g., `[SSD]/mnt/disk/0/ozone`). Even if the underlying physical disks are SSDs, as long as the volume
+configuration does not specify a storage type, the Datanode will treat them as DISK type volumes by default, and no
+configuration changes are needed.
+
+# Protobuf Changes
+
+Major protobuf changes:
+
+hdds.proto
+
+```protobuf
+enum StorageType {
+ DISK_TYPE = 1;
+ SSD_TYPE = 2;
+ ARCHIVE_TYPE = 3;
+ RAM_DISK_TYPE = 4;
+ PROVIDED_TYPE = 5;
+}
+
+enum StorageTierProto {
Review Comment:
A storage policy needs to be mapped to a set of storage types, such as `HOT -> [SSD, SSD, SSD]`. Using storage tiers provides a simpler way to represent a set of storage types, making it more convenient for both coding and management.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
+ Datanode, it will try to use the fallback storage types defined by the SCM container's storage tier. If the storage
+ tier has no defined fallback storage types, or no Datanodes matching the fallback storage types can be found, the
+ operation fails.
+
+## DiskBalancerService
+
+- DiskBalancerService calculates space usage and migrates containers between volumes of the same storage type within a
+ Datanode.
+
+# Configurations
+
+New configuration keys introduced by the storage policy feature:
+
+- `ozone.default.storagepolicy`
+ Default value: WARM. The default storage policy for writing keys when neither the key nor the bucket has a storage
+ policy set.
+
+- `ozone.default.storageTier`
+ Default value: DISK. The default storage tier.
+
+- `ozone.s3.default.storagepolicy`
+ Default value: STANDARD_IA. The default S3 storage class when StorageClass is not specified in the S3 request and
+ the bucket does not have a storage policy set. STANDARD_IA maps to Ozone storage policy WARM.
+
+- `hdds.datanode.default.storagetype`
+ Default value: DISK. The default storage type used when creating a container replica on a Datanode without a
+ specified storage type. This typically applies to write requests from older clients that do not support the storage
+ policy feature.
+
+- `ozone.scm.container.allow.null.storage.tier`
+ Default value: false. Whether to allow selecting containers with a null storage tier during container allocation.
+ This can be used during the upgrade period to allow new keys to be written to existing containers that do not yet
+ have a storage tier set.
+
+For a pure SSD cluster (where all Datanode volumes are configured with SSD storage type), the following configurations
+should be updated accordingly:
+
+- `ozone.default.storagepolicy` should be set to HOT (Hot maps to SSD storage tier).
+- `ozone.default.storageTier` should be set to SSD.
+- `ozone.s3.default.storagepolicy` should be set to STANDARD (STANDARD maps to Hot storage policy).
+- `hdds.datanode.default.storagetype` should be set to SSD.
+
+Otherwise, when writing a key without specifying a storage policy, the system will default to the WARM storage policy,
+which corresponds to the DISK storage tier. Since WARM does not define a fallback tier (its fallback is EMPTY), SCM
+cannot fall back to another tier, and the write will fail because no DISK type volumes are available in the cluster.
+
+Note that this situation only occurs when the volumes are explicitly configured as SSD in the `hdds.datanode.dir`
+configuration (e.g., `[SSD]/mnt/disk/0/ozone`). Even if the underlying physical disks are SSDs, as long as the volume
+configuration does not specify a storage type, the Datanode will treat them as DISK type volumes by default, and no
+configuration changes are needed.
+
+# Protobuf Changes
+
+Major protobuf changes:
+
+hdds.proto
+
+```protobuf
+enum StorageType {
+ DISK_TYPE = 1;
+ SSD_TYPE = 2;
+ ARCHIVE_TYPE = 3;
+ RAM_DISK_TYPE = 4;
+ PROVIDED_TYPE = 5;
+}
+
+enum StorageTierProto {
+ UNKNOWN_TIER = 0;
+ DISK_TIER = 1;
+ SSD_TIER = 2;
+ ARCHIVE_TIER = 3;
+}
+
+enum StoragePolicyProto {
+ UNKNOWN_POLICY = 0;
+ HOT = 1;
+ WARM = 2;
+ COLD = 3;
+}
+
+enum S3StorageClassProto {
+ UNKNOWN_STORAGE_CLASS = 0;
+ STANDARD = 1;
+ STANDARD_IA = 2;
+ GLACIER = 3;
+ DEEP_ARCHIVE = 4;
+}
+
+message Pipeline {
+ //...
+ repeated StorageTierProto supportedStorageTier = 1;
+}
+
+message ContainerInfoProto {
+ //...
+ optional StorageTierProto storageTier = 1;
+}
+```
+
+OmClientProtocol.proto
+
+```protobuf
+message BucketInfo {
+ //...
+ optional hadoop.hdds.StoragePolicyProto storagePolicy = 1;
+ optional bool allowFallbackStoragePolicy = 1;
+}
+
+message KeyInfo {
+ //...
+ optional hdds.StoragePolicyProto storagePolicy = 1;
+}
+```
+
+ScmServerDatanodeHeartbeatProtocol.proto
+
+```protobuf
+message StorageReportProto {
+ //...
+ optional StorageType storageTypeProto = 1 [default = DISK_TYPE];
+}
+
+message ContainerReplicaProto {
+ //...
+ optional StorageType storageType = 1;
+ optional StorageType VolumeStorageType = 1;
+}
+```
+
+# HDFS Compatibility
+
+HDFS storage policy-related commands are not supported.
+
+Specifying a storage policy when creating a file using the HDFS Filesystem interface is not supported, as the Hadoop
+FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy through the create method.
+
+# Backward Compatibility
+
+This section describes the backward compatibility for data created before the storage policy feature is introduced.
Review Comment:
Our upgrade sequence is SCM -> OM -> Datanode.
SCM and OM: Since SCM and OM operate in HA mode, we must first upgrade the two followers, transfer leadership to the upgraded node, and then upgrade the last remaining old leader.
Datanode: Before upgrading a Datanode, it must be set to maintenance mode. Once the Datanode enters maintenance mode, all its pipelines are shut down, and the Datanode can be restarted immediately.
The upgrade process is transparent to clients.
The above upgrade process applies to most version upgrades, including storage policy version upgrades. Upgrading the storage policy version generally does not require any special steps.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
Review Comment:
S3
You can use the standard AWS S3 command or SDK; specify the storage-class when setting the put key.
Ozone
The Ozone API has added a `storagepolicy` parameter.
However, the more common approach is to create a bucket and set a storage policy; then, all writes to that bucket will be subject to the specified storage policy.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
+ Datanode, it will try to use the fallback storage types defined by the SCM container's storage tier. If the storage
+ tier has no defined fallback storage types, or no Datanodes matching the fallback storage types can be found, the
+ operation fails.
+
+## DiskBalancerService
+
+- DiskBalancerService calculates space usage and migrates containers between volumes of the same storage type within a
+ Datanode.
+
+# Configurations
+
+New configuration keys introduced by the storage policy feature:
+
+- `ozone.default.storagepolicy`
+ Default value: WARM. The default storage policy for writing keys when neither the key nor the bucket has a storage
+ policy set.
+
+- `ozone.default.storageTier`
+ Default value: DISK. The default storage tier.
+
+- `ozone.s3.default.storagepolicy`
+ Default value: STANDARD_IA. The default S3 storage class when StorageClass is not specified in the S3 request and
+ the bucket does not have a storage policy set. STANDARD_IA maps to Ozone storage policy WARM.
+
+- `hdds.datanode.default.storagetype`
+ Default value: DISK. The default storage type used when creating a container replica on a Datanode without a
+ specified storage type. This typically applies to write requests from older clients that do not support the storage
+ policy feature.
+
+- `ozone.scm.container.allow.null.storage.tier`
+ Default value: false. Whether to allow selecting containers with a null storage tier during container allocation.
+ This can be used during the upgrade period to allow new keys to be written to existing containers that do not yet
+ have a storage tier set.
+
+For a pure SSD cluster (where all Datanode volumes are configured with SSD storage type), the following configurations
+should be updated accordingly:
+
+- `ozone.default.storagepolicy` should be set to HOT (Hot maps to SSD storage tier).
+- `ozone.default.storageTier` should be set to SSD.
+- `ozone.s3.default.storagepolicy` should be set to STANDARD (STANDARD maps to Hot storage policy).
+- `hdds.datanode.default.storagetype` should be set to SSD.
+
+Otherwise, when writing a key without specifying a storage policy, the system will default to the WARM storage policy,
+which corresponds to the DISK storage tier. Since WARM does not define a fallback tier (its fallback is EMPTY), SCM
+cannot fall back to another tier, and the write will fail because no DISK type volumes are available in the cluster.
+
+Note that this situation only occurs when the volumes are explicitly configured as SSD in the `hdds.datanode.dir`
+configuration (e.g., `[SSD]/mnt/disk/0/ozone`). Even if the underlying physical disks are SSDs, as long as the volume
+configuration does not specify a storage type, the Datanode will treat them as DISK type volumes by default, and no
+configuration changes are needed.
+
+# Protobuf Changes
+
+Major protobuf changes:
+
+hdds.proto
+
+```protobuf
+enum StorageType {
+ DISK_TYPE = 1;
+ SSD_TYPE = 2;
+ ARCHIVE_TYPE = 3;
+ RAM_DISK_TYPE = 4;
+ PROVIDED_TYPE = 5;
+}
+
+enum StorageTierProto {
+ UNKNOWN_TIER = 0;
+ DISK_TIER = 1;
+ SSD_TIER = 2;
+ ARCHIVE_TIER = 3;
+}
+
+enum StoragePolicyProto {
+ UNKNOWN_POLICY = 0;
+ HOT = 1;
+ WARM = 2;
+ COLD = 3;
+}
+
+enum S3StorageClassProto {
+ UNKNOWN_STORAGE_CLASS = 0;
+ STANDARD = 1;
+ STANDARD_IA = 2;
+ GLACIER = 3;
+ DEEP_ARCHIVE = 4;
+}
+
+message Pipeline {
+ //...
+ repeated StorageTierProto supportedStorageTier = 1;
Review Comment:
This works exactly the same as the original logic: if the volume changes, SCM will detect it and may recreate the pipeline.
> What will happen if volumes change: volumes crash, or DN adds/removes volumes between heartbeats.
Adding or removing a volume requires a restart.
If a volume is marked as failed or an exception such as Volume DISK_OUT_OF_SPACE occurs during the heartbeat interval, KeyDataStreamOutput.handleException will be applied, causing a block to be reallocated.
> What should operators do with pipelines when DN storage changes (pipelines close/create)
- Changing the storage type of a volume requires restarting the Datanode.
- If the storage type changes, the SCM will update the Pipeline's storage tier field.
- If the SCM assigns a storage type that the Datanode doesn't actually support, a `KeyDataStreamOutput.handleException` will trigger a block reassignment (this is practically impossible, as changing the storage type of a volume requires restarting the Datanode).
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
Review Comment:
> DN may lose volume between pipeline creation and container creation,
If a change in the volume causes a chunk write to fail, the client should attempt the operation again and reassign the block; this does not actually change the existing retry mechanism.
if writeChunk encounter exception, the`KeyDataStreamOutput#handleException` will allocate a new Block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2999375115
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -19,379 +20,588 @@ status: draft
# Terminology
-## Terminology
+## Definitions
- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
-- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
-- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
-- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
-- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
## Storage Policy vs Storage Type vs Storage Tier
-The relation of Storage Policy, Storage Type and Storage Tier
+The relationship between Storage Policy, Storage Type, and Storage Tier:
-- The storage policy is the property of key/bucket/ prefix (Managed by OM);
-- The storage tier is the property of Pipeline and Container (Managed by SCM);
-- The storage type is the property of volume and Container replicas (Managed by DN);
-- Only the storage policy can be modified by the user directly via ozone command;
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
Example:
-For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
# User Scenarios
-- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
-- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
-- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
-- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
-# Current Status
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
-- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
-- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
-# Goal Requirements Specification
+### Storage Tier Map to Storage Type
-### **Support for Storage Policy Writing and Management**
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
-- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
-- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+### Fallback Storage Type for Container Replica Replication/Migration
-### **Support for Data Migration Across Different Storage Policies**
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
-- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
Review Comment:
Got it. I miss that part. Thanks for reminding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3007348520
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
Review Comment:
Yes, I think it's highly needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2996045328
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -19,379 +20,588 @@ status: draft
# Terminology
-## Terminology
+## Definitions
- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
-- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
-- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
-- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
-- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
## Storage Policy vs Storage Type vs Storage Tier
-The relation of Storage Policy, Storage Type and Storage Tier
+The relationship between Storage Policy, Storage Type, and Storage Tier:
-- The storage policy is the property of key/bucket/ prefix (Managed by OM);
-- The storage tier is the property of Pipeline and Container (Managed by SCM);
-- The storage type is the property of volume and Container replicas (Managed by DN);
-- Only the storage policy can be modified by the user directly via ozone command;
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
Example:
-For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
# User Scenarios
-- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
-- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
-- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
-- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
-# Current Status
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
-- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
-- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
-# Goal Requirements Specification
+### Storage Tier Map to Storage Type
-### **Support for Storage Policy Writing and Management**
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
-- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
-- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+### Fallback Storage Type for Container Replica Replication/Migration
-### **Support for Data Migration Across Different Storage Policies**
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
-- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
-### **Adaptation of AWS S3 StorageClass**
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
-- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+### AWS S3 StorageClass
-### **Management and Monitoring Tools**
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
-- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
-- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
-### **Future Enhancements**
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
-- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
-- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
-- **Recon Support**: Enhance Recon to display relevant storage tier information.
+## Component Changes
-# Detailed Requirements Specification
+### Datanode Container Replica
-## Storage Policy and Storage Types
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
-### Supported Storage Types
+### Bucket, Key
-- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
-- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+A storage policy attribute is added to buckets and keys on OM.
-### Supported Storage Policies
+### SCM Container, Pipeline
-Support storage policy: Hot , Warm, Cold
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
-### Storage Policies Map To Storage Tiers
+## Datanode Volume Storage Type
-| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
-| --- | --- | --- |
-| Hot | SSD | DISK |
-| Warm | DISK | none |
-| Cold | ARCHIVE | none |
-- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
-- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
-### Storage Tier Map To Storage Type
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
-| Tier | StorageType of Pipeline | One Replication
-Container Replicas Storage Type | Three replication
-Container Replicas Storage Type | EC
-Container Replicas Storage Type |
-| --- | --- | --- | --- | --- |
-| SSD | SSD | SSD | 3 SSD | n SSD |
-| DISK | DISK | DISK | 3 DISK | n DISK |
-| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+## Writing Keys with Storage Policy
-### Fallback Storage Type For Container replicas Replication/Migration
+### Ozone Filesystem
-| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
-| --- | --- |
-| SSD | DISK |
-| DISK | none |
-| ARCHIVE | none |
-- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
-[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
-## Support for Ozone Storage Policy Writing and Management
+### S3
-### Support storage policy writing
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
-- Support specifying a storage policy when writing a key.
- - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
- - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
- - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
- - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+### Supported APIs
-### Support fallback strategy configuration
+#### Ozone Filesystem
-- fallbackStrategy
- - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
- - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
-### Inheritance of storage policies
+#### S3 Request
-- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
-- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
-[1] Effective storage policy means a non-empty storage policy.
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
-### undefined storage policy
+### Write Key Process
-- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
-- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
-- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
-- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+#### Normal Write (No Fallback)
-### default storage policy
+
-- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
-- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
-- If the user has not configured a default storage policy, the default storage policy should be Warm.
-- The default storage policy can be configured.
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
-## Storage policy management
+#### Fallback Write
-### key storage policy
+
-- Support setting and unsetting the storage policy for keys.
- - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
-- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
-### Bucket storage policy management
+### Datanode Container Replica Creation
-- Support setting and unsetting storage policies for buckets.
- - After unsetting the storage policy, the storage policy is the default storage policy.
-- Support specifying a storage policy when creating a bucket.
- - If no storage policy is specified, the storage policy is the default storage policy.
-- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
-### Prefix management
+### SCM Pipeline Creation and Selection
-- Support creating, deleting, setting, getting, and listing prefixes.
-- The display of the prefix storage policy should display whether the storage policy is the default storage policy
-- A prefix can only have one type of policy.
-- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
-### Support for persistent storage of storage policy changes:
-- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
-### FSO type buckets:
+## Storage Policy Update
-- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
-- Do not support setting storage policies for directory-type keys.
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
-## Adaptation of AWS S3
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
-### Adaptation of AWS S3 StorageClass
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
-Not all the StorageClass will be support by the Ozone
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
-A possible solution
+## Storage Policy Display
-| AWS S3 StorageClass | Ozone StoragePolicy |
-| --- | --- |
-| STANDARD | Hot |
-| STANDARD_IA | Warm |
-| GLACIER | COLD |
+Display the bucket storage policy attribute:
-> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+- `ozone sh bucket info`
-> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+Display the key storage policy attribute:
-### Adaptation of AWS S3 Related API
+- `ozone sh key info`
-refer to
-[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
-- PutObject:
- - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
-- CopyObject:
- - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
-- Multipart Upload
- - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
-- GetObject operation:
- - Return the current storage policy of the object in the GetObject response.
-- HeadObject:
- - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
-- ListObjects:
- - Include the storage policy (StorageClass) information of each object in the ListObjects response.
-- Bucket StorageClass Lifecycle Rules:
- - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
-- ~~RestoreObject operation: [Not Supported]~~
+- `ozone admin container info`
-## Support for Storage Policy Management Commands/Metrics
+List containers filtered by storage tier:
-Lists commands that need to be added/adapted to storage policies, but may not be all commands
+- `ozone admin container list --storage-tier=<storageTierStr>`
-- Storage policy management
- - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
- - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
- - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
- - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
- - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
- - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
- - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
-- `ozone admin container info`, supports displaying the storage policy of Containers.
-- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
-- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
-- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
-- Datanode volume related Metrics support displaying the storage policies of volumes.
-- SCM supports displaying the data space information of different storage policies in the cluster.
+Display the storage tiers supported by pipelines associated with Datanodes:
-## Storage Policy Satisfier Service
+- `ozone admin datanode list`
-- Support data migration across different storage policies.
- - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
-- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
-- Responsible for managing storage policy-related migration work.
-- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+Display Datanode usage information by storage type:
-## Permissions Management
+- `ozone admin datanode usageinfo`
-- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+Display cluster-wide space usage information by storage type:
-## Storage Policy Supported Replicas Types
+- `ozone admin storagepolicy usageinfo`
-- Support replica type storage policies.
-- Support EC type storage policies.
- - EC types cannot support some storage policies that mix different storage media.
+Check whether a key's storage policy is satisfied:
-## Storage Policy Supported Bucket Types
+- `ozone admin storagepolicy check`
-- Support all types of buckets.
+## Container Balancer
-## Compatible HDFS Commands
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
-HDFS storage policy-related commands are not supported.
+## ReplicationManager
-Do not support specified storage policy when creating a file using HDFS Filesystem interface, the Hadoop FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy by Create method.
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
+ Datanode, it will try to use the fallback storage types defined by the SCM container's storage tier. If the storage
+ tier has no defined fallback storage types, or no Datanodes matching the fallback storage types can be found, the
+ operation fails.
-- Ozone needs to record the access frequency of keys/buckets.
+## DiskBalancerService
-## Recon Side Feature
+- DiskBalancerService calculates space usage and migrates containers between volumes of the same storage type within a
+ Datanode.
-Support displaying relevant storage tier information in Recon. Need to design more detailed. // TODO
+# Configurations
-# Architecture Design
+New configuration keys introduced by the storage policy feature:
-This section aims to describe the architecture and design concepts of the system without delving into specific implementation details. Its purpose is to validate the feasibility and rationality of the technical route, as well as to assess the workload and technical dependencies of related tasks.
+- `ozone.default.storagepolicy`
+ Default value: WARM. The default storage policy for writing keys when neither the key nor the bucket has a storage
+ policy set.
-## Writing Key Process
+- `ozone.default.storageTier`
+ Default value: DISK. The default storage tier.
-Writing data to the specified tier:
+- `ozone.s3.default.storagepolicy`
+ Default value: STANDARD_IA. The default S3 storage class when StorageClass is not specified in the S3 request and
+ the bucket does not have a storage policy set. STANDARD_IA maps to Ozone storage policy WARM.
-- When a user creates a key, they set the storage policy and fallbackStrategy parameters and send the request to the OM:
- - CreateKey
- - CreateFile
- - CreateMultipartKey
-- Community Ozone:
- - CreateStreamKey
- - CreateStreamFile
- - CreateMultipartStreamKey
-- OM adds the storage policy information to the allocateBlock request sent to the SCM. The actual storage strategy should be calculated based on the storage strategy requirements above.
-- SCM, upon receiving the request, selects the Pipeline based on the storage policy and allocates a Container.
- - The storage type of each Container replicas should be added to for the Datanodes in the Pipeline, just like the replicaIndex for the EC key.
- - If the expected storage tier Pipeline cannot be found, attempt to find the fallback tier defined by the storage policy if the fallbackStrategy is allowed. If it cannot be found, the creation process fails.
- - If the SCM needs to create a new Container for a specific storage tier, SCM needs to record the Container’s storage tier for the Container. So the ContainerData should add a storage tier field.
-- After receiving the Pipeline with the specified storage type information, the client writes the Chunk replica with the storage type information.
-- Datanode writes data to the specified Container replica or creates a Container replica in the specified volume based on the storage type .
-- OMKeyInfo needs to persist in the storage policy field, if the storage policy does not be specified, the storage policy should be null.
+- `hdds.datanode.default.storagetype`
+ Default value: DISK. The default storage type used when creating a container replica on a Datanode without a
+ specified storage type. This typically applies to write requests from older clients that do not support the storage
+ policy feature.
-## DN Container Replica Creation
+- `ozone.scm.container.allow.null.storage.tier`
+ Default value: false. Whether to allow selecting containers with a null storage tier during container allocation.
+ This can be used during the upgrade period to allow new keys to be written to existing containers that do not yet
+ have a storage tier set.
+
+For a pure SSD cluster (where all Datanode volumes are configured with SSD storage type), the following configurations
+should be updated accordingly:
+
+- `ozone.default.storagepolicy` should be set to HOT (Hot maps to SSD storage tier).
+- `ozone.default.storageTier` should be set to SSD.
+- `ozone.s3.default.storagepolicy` should be set to STANDARD (STANDARD maps to Hot storage policy).
+- `hdds.datanode.default.storagetype` should be set to SSD.
+
+Otherwise, when writing a key without specifying a storage policy, the system will default to the WARM storage policy,
+which corresponds to the DISK storage tier. Since WARM does not define a fallback tier (its fallback is EMPTY), SCM
+cannot fall back to another tier, and the write will fail because no DISK type volumes are available in the cluster.
+
+Note that this situation only occurs when the volumes are explicitly configured as SSD in the `hdds.datanode.dir`
+configuration (e.g., `[SSD]/mnt/disk/0/ozone`). Even if the underlying physical disks are SSDs, as long as the volume
+configuration does not specify a storage type, the Datanode will treat them as DISK type volumes by default, and no
+configuration changes are needed.
+
+# Protobuf Changes
+
+Major protobuf changes:
+
+hdds.proto
+
+```protobuf
+enum StorageType {
+ DISK_TYPE = 1;
+ SSD_TYPE = 2;
+ ARCHIVE_TYPE = 3;
+ RAM_DISK_TYPE = 4;
+ PROVIDED_TYPE = 5;
+}
+
+enum StorageTierProto {
+ UNKNOWN_TIER = 0;
Review Comment:
Since Proto3 recommends this, Ozone may be upgraded to Proto3 in the future.
https://protobuf.dev/programming-guides/proto3/#default
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -19,379 +20,588 @@ status: draft
# Terminology
-## Terminology
+## Definitions
- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
-- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
-- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
-- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
-- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
## Storage Policy vs Storage Type vs Storage Tier
-The relation of Storage Policy, Storage Type and Storage Tier
+The relationship between Storage Policy, Storage Type, and Storage Tier:
-- The storage policy is the property of key/bucket/ prefix (Managed by OM);
-- The storage tier is the property of Pipeline and Container (Managed by SCM);
-- The storage type is the property of volume and Container replicas (Managed by DN);
-- Only the storage policy can be modified by the user directly via ozone command;
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
Example:
-For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
# User Scenarios
-- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
-- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
-- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
-- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
-# Current Status
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
-- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
-- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
-# Goal Requirements Specification
+### Storage Tier Map to Storage Type
-### **Support for Storage Policy Writing and Management**
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
-- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
-- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+### Fallback Storage Type for Container Replica Replication/Migration
-### **Support for Data Migration Across Different Storage Policies**
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
-- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
-### **Adaptation of AWS S3 StorageClass**
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
-- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+### AWS S3 StorageClass
-### **Management and Monitoring Tools**
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
-- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
-- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
-### **Future Enhancements**
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
-- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
-- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
-- **Recon Support**: Enhance Recon to display relevant storage tier information.
+## Component Changes
-# Detailed Requirements Specification
+### Datanode Container Replica
-## Storage Policy and Storage Types
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
-### Supported Storage Types
+### Bucket, Key
-- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
-- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+A storage policy attribute is added to buckets and keys on OM.
-### Supported Storage Policies
+### SCM Container, Pipeline
-Support storage policy: Hot , Warm, Cold
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
-### Storage Policies Map To Storage Tiers
+## Datanode Volume Storage Type
-| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
-| --- | --- | --- |
-| Hot | SSD | DISK |
-| Warm | DISK | none |
-| Cold | ARCHIVE | none |
-- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
-- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
-### Storage Tier Map To Storage Type
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
-| Tier | StorageType of Pipeline | One Replication
-Container Replicas Storage Type | Three replication
-Container Replicas Storage Type | EC
-Container Replicas Storage Type |
-| --- | --- | --- | --- | --- |
-| SSD | SSD | SSD | 3 SSD | n SSD |
-| DISK | DISK | DISK | 3 DISK | n DISK |
-| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+## Writing Keys with Storage Policy
-### Fallback Storage Type For Container replicas Replication/Migration
+### Ozone Filesystem
-| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
-| --- | --- |
-| SSD | DISK |
-| DISK | none |
-| ARCHIVE | none |
-- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
-[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
-## Support for Ozone Storage Policy Writing and Management
+### S3
-### Support storage policy writing
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
-- Support specifying a storage policy when writing a key.
- - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
- - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
- - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
- - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+### Supported APIs
-### Support fallback strategy configuration
+#### Ozone Filesystem
-- fallbackStrategy
- - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
- - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
-### Inheritance of storage policies
+#### S3 Request
-- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
-- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
-[1] Effective storage policy means a non-empty storage policy.
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
-### undefined storage policy
+### Write Key Process
-- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
-- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
-- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
-- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+#### Normal Write (No Fallback)
-### default storage policy
+
-- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
-- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
-- If the user has not configured a default storage policy, the default storage policy should be Warm.
-- The default storage policy can be configured.
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
-## Storage policy management
+#### Fallback Write
-### key storage policy
+
-- Support setting and unsetting the storage policy for keys.
- - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
-- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
-### Bucket storage policy management
+### Datanode Container Replica Creation
-- Support setting and unsetting storage policies for buckets.
- - After unsetting the storage policy, the storage policy is the default storage policy.
-- Support specifying a storage policy when creating a bucket.
- - If no storage policy is specified, the storage policy is the default storage policy.
-- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
-### Prefix management
+### SCM Pipeline Creation and Selection
-- Support creating, deleting, setting, getting, and listing prefixes.
-- The display of the prefix storage policy should display whether the storage policy is the default storage policy
-- A prefix can only have one type of policy.
-- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
-### Support for persistent storage of storage policy changes:
-- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
-### FSO type buckets:
+## Storage Policy Update
-- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
-- Do not support setting storage policies for directory-type keys.
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
-## Adaptation of AWS S3
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
-### Adaptation of AWS S3 StorageClass
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
-Not all the StorageClass will be support by the Ozone
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
-A possible solution
+## Storage Policy Display
-| AWS S3 StorageClass | Ozone StoragePolicy |
-| --- | --- |
-| STANDARD | Hot |
-| STANDARD_IA | Warm |
-| GLACIER | COLD |
+Display the bucket storage policy attribute:
-> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+- `ozone sh bucket info`
-> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+Display the key storage policy attribute:
-### Adaptation of AWS S3 Related API
+- `ozone sh key info`
-refer to
-[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
-- PutObject:
- - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
-- CopyObject:
- - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
-- Multipart Upload
- - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
-- GetObject operation:
- - Return the current storage policy of the object in the GetObject response.
-- HeadObject:
- - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
-- ListObjects:
- - Include the storage policy (StorageClass) information of each object in the ListObjects response.
-- Bucket StorageClass Lifecycle Rules:
- - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
-- ~~RestoreObject operation: [Not Supported]~~
+- `ozone admin container info`
-## Support for Storage Policy Management Commands/Metrics
+List containers filtered by storage tier:
-Lists commands that need to be added/adapted to storage policies, but may not be all commands
+- `ozone admin container list --storage-tier=<storageTierStr>`
-- Storage policy management
- - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
- - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
- - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
- - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
- - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
- - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
- - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
-- `ozone admin container info`, supports displaying the storage policy of Containers.
-- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
-- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
-- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
-- Datanode volume related Metrics support displaying the storage policies of volumes.
-- SCM supports displaying the data space information of different storage policies in the cluster.
+Display the storage tiers supported by pipelines associated with Datanodes:
-## Storage Policy Satisfier Service
+- `ozone admin datanode list`
-- Support data migration across different storage policies.
- - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
-- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
-- Responsible for managing storage policy-related migration work.
-- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+Display Datanode usage information by storage type:
-## Permissions Management
+- `ozone admin datanode usageinfo`
-- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+Display cluster-wide space usage information by storage type:
-## Storage Policy Supported Replicas Types
+- `ozone admin storagepolicy usageinfo`
-- Support replica type storage policies.
-- Support EC type storage policies.
- - EC types cannot support some storage policies that mix different storage media.
+Check whether a key's storage policy is satisfied:
-## Storage Policy Supported Bucket Types
+- `ozone admin storagepolicy check`
-- Support all types of buckets.
+## Container Balancer
-## Compatible HDFS Commands
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
-HDFS storage policy-related commands are not supported.
+## ReplicationManager
-Do not support specified storage policy when creating a file using HDFS Filesystem interface, the Hadoop FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy by Create method.
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
+ Datanode, it will try to use the fallback storage types defined by the SCM container's storage tier. If the storage
+ tier has no defined fallback storage types, or no Datanodes matching the fallback storage types can be found, the
+ operation fails.
-- Ozone needs to record the access frequency of keys/buckets.
+## DiskBalancerService
-## Recon Side Feature
+- DiskBalancerService calculates space usage and migrates containers between volumes of the same storage type within a
+ Datanode.
-Support displaying relevant storage tier information in Recon. Need to design more detailed. // TODO
+# Configurations
-# Architecture Design
+New configuration keys introduced by the storage policy feature:
-This section aims to describe the architecture and design concepts of the system without delving into specific implementation details. Its purpose is to validate the feasibility and rationality of the technical route, as well as to assess the workload and technical dependencies of related tasks.
+- `ozone.default.storagepolicy`
+ Default value: WARM. The default storage policy for writing keys when neither the key nor the bucket has a storage
+ policy set.
-## Writing Key Process
+- `ozone.default.storageTier`
+ Default value: DISK. The default storage tier.
-Writing data to the specified tier:
+- `ozone.s3.default.storagepolicy`
+ Default value: STANDARD_IA. The default S3 storage class when StorageClass is not specified in the S3 request and
+ the bucket does not have a storage policy set. STANDARD_IA maps to Ozone storage policy WARM.
-- When a user creates a key, they set the storage policy and fallbackStrategy parameters and send the request to the OM:
- - CreateKey
- - CreateFile
- - CreateMultipartKey
-- Community Ozone:
- - CreateStreamKey
- - CreateStreamFile
- - CreateMultipartStreamKey
-- OM adds the storage policy information to the allocateBlock request sent to the SCM. The actual storage strategy should be calculated based on the storage strategy requirements above.
-- SCM, upon receiving the request, selects the Pipeline based on the storage policy and allocates a Container.
- - The storage type of each Container replicas should be added to for the Datanodes in the Pipeline, just like the replicaIndex for the EC key.
- - If the expected storage tier Pipeline cannot be found, attempt to find the fallback tier defined by the storage policy if the fallbackStrategy is allowed. If it cannot be found, the creation process fails.
- - If the SCM needs to create a new Container for a specific storage tier, SCM needs to record the Container’s storage tier for the Container. So the ContainerData should add a storage tier field.
-- After receiving the Pipeline with the specified storage type information, the client writes the Chunk replica with the storage type information.
-- Datanode writes data to the specified Container replica or creates a Container replica in the specified volume based on the storage type .
-- OMKeyInfo needs to persist in the storage policy field, if the storage policy does not be specified, the storage policy should be null.
+- `hdds.datanode.default.storagetype`
+ Default value: DISK. The default storage type used when creating a container replica on a Datanode without a
+ specified storage type. This typically applies to write requests from older clients that do not support the storage
+ policy feature.
-## DN Container Replica Creation
+- `ozone.scm.container.allow.null.storage.tier`
+ Default value: false. Whether to allow selecting containers with a null storage tier during container allocation.
+ This can be used during the upgrade period to allow new keys to be written to existing containers that do not yet
Review Comment:
This is because if the tier of a container created by SCM before the upgrade is null, and if these containers are not selected after the upgrade, no new data will be written to them, causing them to remain open (In the normal process, a container must be filled before it can be closed).
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -19,379 +20,588 @@ status: draft
# Terminology
-## Terminology
+## Definitions
- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
-- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
-- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
-- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
-- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
## Storage Policy vs Storage Type vs Storage Tier
-The relation of Storage Policy, Storage Type and Storage Tier
+The relationship between Storage Policy, Storage Type, and Storage Tier:
-- The storage policy is the property of key/bucket/ prefix (Managed by OM);
-- The storage tier is the property of Pipeline and Container (Managed by SCM);
-- The storage type is the property of volume and Container replicas (Managed by DN);
-- Only the storage policy can be modified by the user directly via ozone command;
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
Example:
-For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
# User Scenarios
-- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
-- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
-- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
-- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
-# Current Status
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
-- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
-- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
-# Goal Requirements Specification
+### Storage Tier Map to Storage Type
-### **Support for Storage Policy Writing and Management**
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
-- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
-- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+### Fallback Storage Type for Container Replica Replication/Migration
-### **Support for Data Migration Across Different Storage Policies**
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
-- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
Review Comment:
Yes, ReplicationManager's `containerCheckChain` first handles exceptions such as `UNDER_REPLICATED`, `MIS_REPLICATED`, and `OVER_REPLICATED` (which may involve a fallback storage type). Once the container's replication status is no longer in one of these states, ReplicationManager will check `MIS_STORAGE_TYPE_WITH_VOLUME` and `MIS_STORAGE_TYPE_WITH_CONTAINER` and will migrate those Container Replica to their storage type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2875955485
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
Review Comment:
Does LifecycleService/Jobworker have information about all keys that don't comply with the storage policy? Does it run the migration key by key iteratively?
At the moment, in the patch, I couldn't see any API that allows collecting information about all keys that don't satisfy the storage policy. Also, is there any plan to migrate bulk file migration rather than key by key using `rewriteKey`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3850649811
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
Review Comment:
The actual purpose of DEEP_ARCHIVE is to represent EC replication. When the Action Transition in Lifecycle is set to the storageClass DEEP_ARCHIVE, the key will be migrated to an EC replica.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
Review Comment:
Fallback is enabled by default. If Fallback is disabled, write operations will fail immediately when space runs out; otherwise, Fallback will be implicit, and implicit Fallback may not be necessary in all scenarios.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
Review Comment:
In the current implementation, a pipeline supports only one storage tier, as having multiple storage tiers may cause issues. For more information, see
https://github.com/apache/ozone/pull/10573#issuecomment-4886688548
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
Review Comment:
Due to the current design, only newly created containers have a storage tier filed. For these containers, the container replicas on the data nodes will be migrated or replicated to matching volume based on the storage tier specified in the SCM.
For existing containers, the storage tier in the SCM is null. The behavior of these containers will remain the same as before, and the volume’s storage type will not be considered during migration.
If you want the volume’s storage type to be taken into account during the migration of existing containers (for example, to prevent them from being migrated to SSDs), you need to set a storage tier for the container in SCM (this can also be achieved through configuration, such as treating a null storage tier as a “null disk” storage tier); the command line is a more flexible approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-2976778259
@vtutrinov Thanks for the reminder. I have attached https://issues.apache.org/jira/secure/attachment/13077025/storage-policy-diff.tar.gz for the list of diffs of the storage policy integration.
Please be reminded to attribute @xichen01 for any commits generated from these diffs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r1692365363
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
Review Comment:
Nit: The table does not seem to be rendered properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-2251173313
cc @sodonnel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3007348520
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
Review Comment:
Yes, I think it's highly required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2906025795
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
Review Comment:
Thanks for the explanation. Do we have any discussions on whether the Jobworker should be a standalone component or integrated into an existing components? Also, is it a scheduler or the actual worker doing the heavy lifting?
In my opinion, throttling the I/O directly would be more effective and straightforward than limiting the number of worker threads. It gives us more direct control over the resource of migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4378706326
The new HDDS-11233 branch https://github.com/apache/ozone/tree/HDDS-11233
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-5075803259
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3764943896
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
Review Comment:
Will it be better that always allow the fallback, since if the fallback is not allowed, the write will fail. It's better to finish the write on a degraded media, instead of fail the whole write, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-5407631496
> @xichen01 , thanks for sharing the design and working on this feature. Have we considered the future expand of storage policy, for example, to support HDFS like ONE_SSD policy, which has one replica on SSD, and other replicas on HDDS, or WARN policy, which has one replica on HDD and other replicas on ARCHIVE? If yes, what will be the change of the storage policy, and storage tier? If not, is it because this usage is not that popular in reality?
>
> One more question about the FSO bucket, I'm not sure if it's already covered in the design, that if a storage policy is set a directory, will all the new children use this policy if policy is not specified during children creation? Or Children will use bucket's policy?
If you are adding a Uniform storage tier, you only need to add the corresponding enum to `StorageTier`.
If you are adding a non-Uniform storage tier such as `ONE_SSD`, some adaptation is required, as the current implementation is based on the Uniform storage tier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3851029531
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
Review Comment:
In ReplicationManager, whenever a new Container replica is added, a fallback is attempted if the primary storage type out of space.
Containerbalancer does not perform a fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-2955438634
@ivandika3 thanks for the response!
Can we glance at the implementation as the first phase (maybe in a custom feature branch)? Or are there too many private details?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "vtutrinov (via GitHub)" <gi...@apache.org>.
vtutrinov commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-2955514606
@ivandika3 it would be great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2100544578
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
Review Comment:
Storage Tier is more like the `ReplicationConfig`, will be a independent fields in `ContainerInfo` and `Pipeline`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-3519269594
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4379351270
@ivandika3 @greenwich @chungen0126
https://github.com/apache/ozone/pull/10191Please help to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4435956576
@ivandika3, @xichen01, as we have HDDS-15239 merged and a new branch created. Are we ready to move forward?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2993177512
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -19,379 +20,588 @@ status: draft
# Terminology
-## Terminology
+## Definitions
- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
-- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
-- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
-- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
-- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
## Storage Policy vs Storage Type vs Storage Tier
-The relation of Storage Policy, Storage Type and Storage Tier
+The relationship between Storage Policy, Storage Type, and Storage Tier:
-- The storage policy is the property of key/bucket/ prefix (Managed by OM);
-- The storage tier is the property of Pipeline and Container (Managed by SCM);
-- The storage type is the property of volume and Container replicas (Managed by DN);
-- Only the storage policy can be modified by the user directly via ozone command;
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
Example:
-For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
# User Scenarios
-- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
-- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
-- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
-- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
-# Current Status
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
-- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
-- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
-# Goal Requirements Specification
+### Storage Tier Map to Storage Type
-### **Support for Storage Policy Writing and Management**
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
-- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
-- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+### Fallback Storage Type for Container Replica Replication/Migration
-### **Support for Data Migration Across Different Storage Policies**
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
-- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
Review Comment:
During the container replica replication or migration process, I think we should prioritize migrating those fallbacked data back to their storage type once there is enough space.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -19,379 +20,588 @@ status: draft
# Terminology
-## Terminology
+## Definitions
- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
-- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
-- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
-- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
-- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
## Storage Policy vs Storage Type vs Storage Tier
-The relation of Storage Policy, Storage Type and Storage Tier
+The relationship between Storage Policy, Storage Type, and Storage Tier:
-- The storage policy is the property of key/bucket/ prefix (Managed by OM);
-- The storage tier is the property of Pipeline and Container (Managed by SCM);
-- The storage type is the property of volume and Container replicas (Managed by DN);
-- Only the storage policy can be modified by the user directly via ozone command;
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
Example:
-For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
# User Scenarios
-- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
-- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
-- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
-- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
-# Current Status
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
-- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
-- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
-# Goal Requirements Specification
+### Storage Tier Map to Storage Type
-### **Support for Storage Policy Writing and Management**
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
-- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
-- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+### Fallback Storage Type for Container Replica Replication/Migration
-### **Support for Data Migration Across Different Storage Policies**
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
-- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
-### **Adaptation of AWS S3 StorageClass**
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
-- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+### AWS S3 StorageClass
-### **Management and Monitoring Tools**
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
-- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
-- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
-### **Future Enhancements**
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
-- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
-- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
-- **Recon Support**: Enhance Recon to display relevant storage tier information.
+## Component Changes
-# Detailed Requirements Specification
+### Datanode Container Replica
-## Storage Policy and Storage Types
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
-### Supported Storage Types
+### Bucket, Key
-- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
-- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+A storage policy attribute is added to buckets and keys on OM.
-### Supported Storage Policies
+### SCM Container, Pipeline
-Support storage policy: Hot , Warm, Cold
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
-### Storage Policies Map To Storage Tiers
+## Datanode Volume Storage Type
-| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
-| --- | --- | --- |
-| Hot | SSD | DISK |
-| Warm | DISK | none |
-| Cold | ARCHIVE | none |
-- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
-- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
-### Storage Tier Map To Storage Type
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
-| Tier | StorageType of Pipeline | One Replication
-Container Replicas Storage Type | Three replication
-Container Replicas Storage Type | EC
-Container Replicas Storage Type |
-| --- | --- | --- | --- | --- |
-| SSD | SSD | SSD | 3 SSD | n SSD |
-| DISK | DISK | DISK | 3 DISK | n DISK |
-| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+## Writing Keys with Storage Policy
-### Fallback Storage Type For Container replicas Replication/Migration
+### Ozone Filesystem
-| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
-| --- | --- |
-| SSD | DISK |
-| DISK | none |
-| ARCHIVE | none |
-- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
-[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
-## Support for Ozone Storage Policy Writing and Management
+### S3
-### Support storage policy writing
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
-- Support specifying a storage policy when writing a key.
- - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
- - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
- - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
- - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+### Supported APIs
-### Support fallback strategy configuration
+#### Ozone Filesystem
-- fallbackStrategy
- - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
- - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
-### Inheritance of storage policies
+#### S3 Request
-- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
-- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
-[1] Effective storage policy means a non-empty storage policy.
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
-### undefined storage policy
+### Write Key Process
-- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
-- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
-- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
-- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+#### Normal Write (No Fallback)
-### default storage policy
+
-- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
-- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
-- If the user has not configured a default storage policy, the default storage policy should be Warm.
-- The default storage policy can be configured.
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
-## Storage policy management
+#### Fallback Write
-### key storage policy
+
-- Support setting and unsetting the storage policy for keys.
- - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
-- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
-### Bucket storage policy management
+### Datanode Container Replica Creation
-- Support setting and unsetting storage policies for buckets.
- - After unsetting the storage policy, the storage policy is the default storage policy.
-- Support specifying a storage policy when creating a bucket.
- - If no storage policy is specified, the storage policy is the default storage policy.
-- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
-### Prefix management
+### SCM Pipeline Creation and Selection
-- Support creating, deleting, setting, getting, and listing prefixes.
-- The display of the prefix storage policy should display whether the storage policy is the default storage policy
-- A prefix can only have one type of policy.
-- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
-### Support for persistent storage of storage policy changes:
-- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
-### FSO type buckets:
+## Storage Policy Update
-- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
-- Do not support setting storage policies for directory-type keys.
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
-## Adaptation of AWS S3
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
-### Adaptation of AWS S3 StorageClass
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
-Not all the StorageClass will be support by the Ozone
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
-A possible solution
+## Storage Policy Display
-| AWS S3 StorageClass | Ozone StoragePolicy |
-| --- | --- |
-| STANDARD | Hot |
-| STANDARD_IA | Warm |
-| GLACIER | COLD |
+Display the bucket storage policy attribute:
-> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+- `ozone sh bucket info`
-> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+Display the key storage policy attribute:
-### Adaptation of AWS S3 Related API
+- `ozone sh key info`
-refer to
-[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
-- PutObject:
- - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
-- CopyObject:
- - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
-- Multipart Upload
- - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
-- GetObject operation:
- - Return the current storage policy of the object in the GetObject response.
-- HeadObject:
- - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
-- ListObjects:
- - Include the storage policy (StorageClass) information of each object in the ListObjects response.
-- Bucket StorageClass Lifecycle Rules:
- - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
-- ~~RestoreObject operation: [Not Supported]~~
+- `ozone admin container info`
-## Support for Storage Policy Management Commands/Metrics
+List containers filtered by storage tier:
-Lists commands that need to be added/adapted to storage policies, but may not be all commands
+- `ozone admin container list --storage-tier=<storageTierStr>`
-- Storage policy management
- - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
- - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
- - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
- - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
- - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
- - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
- - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
-- `ozone admin container info`, supports displaying the storage policy of Containers.
-- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
-- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
-- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
-- Datanode volume related Metrics support displaying the storage policies of volumes.
-- SCM supports displaying the data space information of different storage policies in the cluster.
+Display the storage tiers supported by pipelines associated with Datanodes:
-## Storage Policy Satisfier Service
+- `ozone admin datanode list`
-- Support data migration across different storage policies.
- - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
-- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
-- Responsible for managing storage policy-related migration work.
-- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+Display Datanode usage information by storage type:
-## Permissions Management
+- `ozone admin datanode usageinfo`
-- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+Display cluster-wide space usage information by storage type:
-## Storage Policy Supported Replicas Types
+- `ozone admin storagepolicy usageinfo`
-- Support replica type storage policies.
-- Support EC type storage policies.
- - EC types cannot support some storage policies that mix different storage media.
+Check whether a key's storage policy is satisfied:
-## Storage Policy Supported Bucket Types
+- `ozone admin storagepolicy check`
-- Support all types of buckets.
+## Container Balancer
-## Compatible HDFS Commands
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
-HDFS storage policy-related commands are not supported.
+## ReplicationManager
-Do not support specified storage policy when creating a file using HDFS Filesystem interface, the Hadoop FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy by Create method.
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
+ Datanode, it will try to use the fallback storage types defined by the SCM container's storage tier. If the storage
+ tier has no defined fallback storage types, or no Datanodes matching the fallback storage types can be found, the
+ operation fails.
-- Ozone needs to record the access frequency of keys/buckets.
+## DiskBalancerService
-## Recon Side Feature
+- DiskBalancerService calculates space usage and migrates containers between volumes of the same storage type within a
+ Datanode.
-Support displaying relevant storage tier information in Recon. Need to design more detailed. // TODO
+# Configurations
-# Architecture Design
+New configuration keys introduced by the storage policy feature:
-This section aims to describe the architecture and design concepts of the system without delving into specific implementation details. Its purpose is to validate the feasibility and rationality of the technical route, as well as to assess the workload and technical dependencies of related tasks.
+- `ozone.default.storagepolicy`
+ Default value: WARM. The default storage policy for writing keys when neither the key nor the bucket has a storage
+ policy set.
-## Writing Key Process
+- `ozone.default.storageTier`
+ Default value: DISK. The default storage tier.
-Writing data to the specified tier:
+- `ozone.s3.default.storagepolicy`
+ Default value: STANDARD_IA. The default S3 storage class when StorageClass is not specified in the S3 request and
+ the bucket does not have a storage policy set. STANDARD_IA maps to Ozone storage policy WARM.
-- When a user creates a key, they set the storage policy and fallbackStrategy parameters and send the request to the OM:
- - CreateKey
- - CreateFile
- - CreateMultipartKey
-- Community Ozone:
- - CreateStreamKey
- - CreateStreamFile
- - CreateMultipartStreamKey
-- OM adds the storage policy information to the allocateBlock request sent to the SCM. The actual storage strategy should be calculated based on the storage strategy requirements above.
-- SCM, upon receiving the request, selects the Pipeline based on the storage policy and allocates a Container.
- - The storage type of each Container replicas should be added to for the Datanodes in the Pipeline, just like the replicaIndex for the EC key.
- - If the expected storage tier Pipeline cannot be found, attempt to find the fallback tier defined by the storage policy if the fallbackStrategy is allowed. If it cannot be found, the creation process fails.
- - If the SCM needs to create a new Container for a specific storage tier, SCM needs to record the Container’s storage tier for the Container. So the ContainerData should add a storage tier field.
-- After receiving the Pipeline with the specified storage type information, the client writes the Chunk replica with the storage type information.
-- Datanode writes data to the specified Container replica or creates a Container replica in the specified volume based on the storage type .
-- OMKeyInfo needs to persist in the storage policy field, if the storage policy does not be specified, the storage policy should be null.
+- `hdds.datanode.default.storagetype`
+ Default value: DISK. The default storage type used when creating a container replica on a Datanode without a
+ specified storage type. This typically applies to write requests from older clients that do not support the storage
+ policy feature.
-## DN Container Replica Creation
+- `ozone.scm.container.allow.null.storage.tier`
+ Default value: false. Whether to allow selecting containers with a null storage tier during container allocation.
+ This can be used during the upgrade period to allow new keys to be written to existing containers that do not yet
Review Comment:
Could you please clarify this configuration. I don't understand why we need it during the upgrade period.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -19,379 +20,588 @@ status: draft
# Terminology
-## Terminology
+## Definitions
- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
-- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
-- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
-- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
-- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
## Storage Policy vs Storage Type vs Storage Tier
-The relation of Storage Policy, Storage Type and Storage Tier
+The relationship between Storage Policy, Storage Type, and Storage Tier:
-- The storage policy is the property of key/bucket/ prefix (Managed by OM);
-- The storage tier is the property of Pipeline and Container (Managed by SCM);
-- The storage type is the property of volume and Container replicas (Managed by DN);
-- Only the storage policy can be modified by the user directly via ozone command;
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
Example:
-For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
# User Scenarios
-- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
-- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
-- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
-- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
-# Current Status
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
-- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
-- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
-# Goal Requirements Specification
+### Storage Tier Map to Storage Type
-### **Support for Storage Policy Writing and Management**
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
-- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
-- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+### Fallback Storage Type for Container Replica Replication/Migration
-### **Support for Data Migration Across Different Storage Policies**
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
-- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
-### **Adaptation of AWS S3 StorageClass**
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
-- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+### AWS S3 StorageClass
-### **Management and Monitoring Tools**
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
-- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
-- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
-### **Future Enhancements**
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
-- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
-- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
-- **Recon Support**: Enhance Recon to display relevant storage tier information.
+## Component Changes
-# Detailed Requirements Specification
+### Datanode Container Replica
-## Storage Policy and Storage Types
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
-### Supported Storage Types
+### Bucket, Key
-- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
-- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+A storage policy attribute is added to buckets and keys on OM.
-### Supported Storage Policies
+### SCM Container, Pipeline
-Support storage policy: Hot , Warm, Cold
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
-### Storage Policies Map To Storage Tiers
+## Datanode Volume Storage Type
-| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
-| --- | --- | --- |
-| Hot | SSD | DISK |
-| Warm | DISK | none |
-| Cold | ARCHIVE | none |
-- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
-- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
-### Storage Tier Map To Storage Type
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
-| Tier | StorageType of Pipeline | One Replication
-Container Replicas Storage Type | Three replication
-Container Replicas Storage Type | EC
-Container Replicas Storage Type |
-| --- | --- | --- | --- | --- |
-| SSD | SSD | SSD | 3 SSD | n SSD |
-| DISK | DISK | DISK | 3 DISK | n DISK |
-| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+## Writing Keys with Storage Policy
-### Fallback Storage Type For Container replicas Replication/Migration
+### Ozone Filesystem
-| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
-| --- | --- |
-| SSD | DISK |
-| DISK | none |
-| ARCHIVE | none |
-- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
-[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
-## Support for Ozone Storage Policy Writing and Management
+### S3
-### Support storage policy writing
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
-- Support specifying a storage policy when writing a key.
- - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
- - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
- - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
- - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+### Supported APIs
-### Support fallback strategy configuration
+#### Ozone Filesystem
-- fallbackStrategy
- - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
- - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
-### Inheritance of storage policies
+#### S3 Request
-- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
-- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
-[1] Effective storage policy means a non-empty storage policy.
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
-### undefined storage policy
+### Write Key Process
-- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
-- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
-- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
-- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+#### Normal Write (No Fallback)
-### default storage policy
+
-- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
-- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
-- If the user has not configured a default storage policy, the default storage policy should be Warm.
-- The default storage policy can be configured.
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
-## Storage policy management
+#### Fallback Write
-### key storage policy
+
-- Support setting and unsetting the storage policy for keys.
- - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
-- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
-### Bucket storage policy management
+### Datanode Container Replica Creation
-- Support setting and unsetting storage policies for buckets.
- - After unsetting the storage policy, the storage policy is the default storage policy.
-- Support specifying a storage policy when creating a bucket.
- - If no storage policy is specified, the storage policy is the default storage policy.
-- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
-### Prefix management
+### SCM Pipeline Creation and Selection
-- Support creating, deleting, setting, getting, and listing prefixes.
-- The display of the prefix storage policy should display whether the storage policy is the default storage policy
-- A prefix can only have one type of policy.
-- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
-### Support for persistent storage of storage policy changes:
-- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
-### FSO type buckets:
+## Storage Policy Update
-- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
-- Do not support setting storage policies for directory-type keys.
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
-## Adaptation of AWS S3
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
-### Adaptation of AWS S3 StorageClass
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
-Not all the StorageClass will be support by the Ozone
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
-A possible solution
+## Storage Policy Display
-| AWS S3 StorageClass | Ozone StoragePolicy |
-| --- | --- |
-| STANDARD | Hot |
-| STANDARD_IA | Warm |
-| GLACIER | COLD |
+Display the bucket storage policy attribute:
-> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+- `ozone sh bucket info`
-> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+Display the key storage policy attribute:
-### Adaptation of AWS S3 Related API
+- `ozone sh key info`
-refer to
-[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
-- PutObject:
- - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
-- CopyObject:
- - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
-- Multipart Upload
- - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
-- GetObject operation:
- - Return the current storage policy of the object in the GetObject response.
-- HeadObject:
- - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
-- ListObjects:
- - Include the storage policy (StorageClass) information of each object in the ListObjects response.
-- Bucket StorageClass Lifecycle Rules:
- - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
-- ~~RestoreObject operation: [Not Supported]~~
+- `ozone admin container info`
-## Support for Storage Policy Management Commands/Metrics
+List containers filtered by storage tier:
-Lists commands that need to be added/adapted to storage policies, but may not be all commands
+- `ozone admin container list --storage-tier=<storageTierStr>`
-- Storage policy management
- - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
- - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
- - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
- - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
- - `ozone admin storagepolicies satisfyStoragePolicy status -bucket / -prefix / -key`, checks the migration status of the storage policy.
- - `ozone admin storagepolicies checkSatisfyStoragePolicy -bucket / -prefix / -key`, checks whether the specified resource's storage policy is satisfied.
- - Containers may be migrated to a fallback tier. At this time, the storage policy of the corresponding resource is not satisfied.
-- `ozone admin container info`, supports displaying the storage policy of Containers.
-- `ozone admin datanode usageinfo`, supports displaying space information according to different storage policies.
-- `ozone admin pipeline list`, supports displaying the storage tier supported by Pipelines.
-- `ozone admin pipeline create`, supports creating Pipelines that support specified storage tier.
-- Datanode volume related Metrics support displaying the storage policies of volumes.
-- SCM supports displaying the data space information of different storage policies in the cluster.
+Display the storage tiers supported by pipelines associated with Datanodes:
-## Storage Policy Satisfier Service
+- `ozone admin datanode list`
-- Support data migration across different storage policies.
- - Storage policy migrations are triggered by ozone admin storagepolicies satisfyStoragePolicy. Modifying the storage policy of a bucket/prefix/key does not directly trigger storage policy migrations.
-- Support manually setting storage policies for buckets/prefixes/keys and implementing asynchronous migrations.
-- Responsible for managing storage policy-related migration work.
-- Responsible for responding to user requests and checking whether the specified resources satisfy the storage policy.
+Display Datanode usage information by storage type:
-## Permissions Management
+- `ozone admin datanode usageinfo`
-- Changing the storage policy of buckets/prefixes/keys requires administrator permissions.
+Display cluster-wide space usage information by storage type:
-## Storage Policy Supported Replicas Types
+- `ozone admin storagepolicy usageinfo`
-- Support replica type storage policies.
-- Support EC type storage policies.
- - EC types cannot support some storage policies that mix different storage media.
+Check whether a key's storage policy is satisfied:
-## Storage Policy Supported Bucket Types
+- `ozone admin storagepolicy check`
-- Support all types of buckets.
+## Container Balancer
-## Compatible HDFS Commands
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
-HDFS storage policy-related commands are not supported.
+## ReplicationManager
-Do not support specified storage policy when creating a file using HDFS Filesystem interface, the Hadoop FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy by Create method.
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
+ Datanode, it will try to use the fallback storage types defined by the SCM container's storage tier. If the storage
+ tier has no defined fallback storage types, or no Datanodes matching the fallback storage types can be found, the
+ operation fails.
-- Ozone needs to record the access frequency of keys/buckets.
+## DiskBalancerService
-## Recon Side Feature
+- DiskBalancerService calculates space usage and migrates containers between volumes of the same storage type within a
+ Datanode.
-Support displaying relevant storage tier information in Recon. Need to design more detailed. // TODO
+# Configurations
-# Architecture Design
+New configuration keys introduced by the storage policy feature:
-This section aims to describe the architecture and design concepts of the system without delving into specific implementation details. Its purpose is to validate the feasibility and rationality of the technical route, as well as to assess the workload and technical dependencies of related tasks.
+- `ozone.default.storagepolicy`
+ Default value: WARM. The default storage policy for writing keys when neither the key nor the bucket has a storage
+ policy set.
-## Writing Key Process
+- `ozone.default.storageTier`
+ Default value: DISK. The default storage tier.
-Writing data to the specified tier:
+- `ozone.s3.default.storagepolicy`
+ Default value: STANDARD_IA. The default S3 storage class when StorageClass is not specified in the S3 request and
+ the bucket does not have a storage policy set. STANDARD_IA maps to Ozone storage policy WARM.
-- When a user creates a key, they set the storage policy and fallbackStrategy parameters and send the request to the OM:
- - CreateKey
- - CreateFile
- - CreateMultipartKey
-- Community Ozone:
- - CreateStreamKey
- - CreateStreamFile
- - CreateMultipartStreamKey
-- OM adds the storage policy information to the allocateBlock request sent to the SCM. The actual storage strategy should be calculated based on the storage strategy requirements above.
-- SCM, upon receiving the request, selects the Pipeline based on the storage policy and allocates a Container.
- - The storage type of each Container replicas should be added to for the Datanodes in the Pipeline, just like the replicaIndex for the EC key.
- - If the expected storage tier Pipeline cannot be found, attempt to find the fallback tier defined by the storage policy if the fallbackStrategy is allowed. If it cannot be found, the creation process fails.
- - If the SCM needs to create a new Container for a specific storage tier, SCM needs to record the Container’s storage tier for the Container. So the ContainerData should add a storage tier field.
-- After receiving the Pipeline with the specified storage type information, the client writes the Chunk replica with the storage type information.
-- Datanode writes data to the specified Container replica or creates a Container replica in the specified volume based on the storage type .
-- OMKeyInfo needs to persist in the storage policy field, if the storage policy does not be specified, the storage policy should be null.
+- `hdds.datanode.default.storagetype`
+ Default value: DISK. The default storage type used when creating a container replica on a Datanode without a
+ specified storage type. This typically applies to write requests from older clients that do not support the storage
+ policy feature.
-## DN Container Replica Creation
+- `ozone.scm.container.allow.null.storage.tier`
+ Default value: false. Whether to allow selecting containers with a null storage tier during container allocation.
+ This can be used during the upgrade period to allow new keys to be written to existing containers that do not yet
+ have a storage tier set.
+
+For a pure SSD cluster (where all Datanode volumes are configured with SSD storage type), the following configurations
+should be updated accordingly:
+
+- `ozone.default.storagepolicy` should be set to HOT (Hot maps to SSD storage tier).
+- `ozone.default.storageTier` should be set to SSD.
+- `ozone.s3.default.storagepolicy` should be set to STANDARD (STANDARD maps to Hot storage policy).
+- `hdds.datanode.default.storagetype` should be set to SSD.
+
+Otherwise, when writing a key without specifying a storage policy, the system will default to the WARM storage policy,
+which corresponds to the DISK storage tier. Since WARM does not define a fallback tier (its fallback is EMPTY), SCM
+cannot fall back to another tier, and the write will fail because no DISK type volumes are available in the cluster.
+
+Note that this situation only occurs when the volumes are explicitly configured as SSD in the `hdds.datanode.dir`
+configuration (e.g., `[SSD]/mnt/disk/0/ozone`). Even if the underlying physical disks are SSDs, as long as the volume
+configuration does not specify a storage type, the Datanode will treat them as DISK type volumes by default, and no
+configuration changes are needed.
+
+# Protobuf Changes
+
+Major protobuf changes:
+
+hdds.proto
+
+```protobuf
+enum StorageType {
+ DISK_TYPE = 1;
+ SSD_TYPE = 2;
+ ARCHIVE_TYPE = 3;
+ RAM_DISK_TYPE = 4;
+ PROVIDED_TYPE = 5;
+}
+
+enum StorageTierProto {
+ UNKNOWN_TIER = 0;
Review Comment:
Why do enums start at 0 by default? This differs from our current Protobuf implementation in Ozone. Is this a new standard?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-3955939286
@xichen01, Thanks for the update. That's great news indeed! In that case, there is no point for me to invent a wheel. I had a quick look at the patch you submitted earlier, and it looks comprehensive. I will have a detailed look today.
Would it be possible to move forward with that PR (and merge it, or at least create a working branch and sync it with master)? Also, as I am very interested in that feature, I can provide assistance on my end.
cc: @ivandika3 @chungen0126
Please let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on PR #6989:
URL: https://github.com/apache/ozone/pull/6989#issuecomment-4234740368
@ivandika3, Have you managed to backport the patch as per the discussion here: https://github.com/apache/ozone/pull/6989#issuecomment-2953554337?
Any help required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "greenwich (via GitHub)" <gi...@apache.org>.
greenwich commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2978734922
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
Review Comment:
As I understand the policy resolution is `key → bucket → config`, any plans for ``key → prefix → bucket → config`?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
Review Comment:
Is there a prefix-level policy support?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
+ Datanode, it will try to use the fallback storage types defined by the SCM container's storage tier. If the storage
+ tier has no defined fallback storage types, or no Datanodes matching the fallback storage types can be found, the
+ operation fails.
+
+## DiskBalancerService
+
+- DiskBalancerService calculates space usage and migrates containers between volumes of the same storage type within a
+ Datanode.
+
+# Configurations
+
+New configuration keys introduced by the storage policy feature:
+
+- `ozone.default.storagepolicy`
+ Default value: WARM. The default storage policy for writing keys when neither the key nor the bucket has a storage
+ policy set.
+
+- `ozone.default.storageTier`
+ Default value: DISK. The default storage tier.
+
+- `ozone.s3.default.storagepolicy`
+ Default value: STANDARD_IA. The default S3 storage class when StorageClass is not specified in the S3 request and
+ the bucket does not have a storage policy set. STANDARD_IA maps to Ozone storage policy WARM.
+
+- `hdds.datanode.default.storagetype`
+ Default value: DISK. The default storage type used when creating a container replica on a Datanode without a
+ specified storage type. This typically applies to write requests from older clients that do not support the storage
+ policy feature.
+
+- `ozone.scm.container.allow.null.storage.tier`
+ Default value: false. Whether to allow selecting containers with a null storage tier during container allocation.
+ This can be used during the upgrade period to allow new keys to be written to existing containers that do not yet
+ have a storage tier set.
+
+For a pure SSD cluster (where all Datanode volumes are configured with SSD storage type), the following configurations
+should be updated accordingly:
+
+- `ozone.default.storagepolicy` should be set to HOT (Hot maps to SSD storage tier).
+- `ozone.default.storageTier` should be set to SSD.
+- `ozone.s3.default.storagepolicy` should be set to STANDARD (STANDARD maps to Hot storage policy).
+- `hdds.datanode.default.storagetype` should be set to SSD.
+
+Otherwise, when writing a key without specifying a storage policy, the system will default to the WARM storage policy,
+which corresponds to the DISK storage tier. Since WARM does not define a fallback tier (its fallback is EMPTY), SCM
+cannot fall back to another tier, and the write will fail because no DISK type volumes are available in the cluster.
+
+Note that this situation only occurs when the volumes are explicitly configured as SSD in the `hdds.datanode.dir`
+configuration (e.g., `[SSD]/mnt/disk/0/ozone`). Even if the underlying physical disks are SSDs, as long as the volume
+configuration does not specify a storage type, the Datanode will treat them as DISK type volumes by default, and no
+configuration changes are needed.
+
+# Protobuf Changes
+
+Major protobuf changes:
+
+hdds.proto
+
+```protobuf
+enum StorageType {
+ DISK_TYPE = 1;
+ SSD_TYPE = 2;
+ ARCHIVE_TYPE = 3;
+ RAM_DISK_TYPE = 4;
+ PROVIDED_TYPE = 5;
+}
+
+enum StorageTierProto {
+ UNKNOWN_TIER = 0;
+ DISK_TIER = 1;
+ SSD_TIER = 2;
+ ARCHIVE_TIER = 3;
+}
+
+enum StoragePolicyProto {
+ UNKNOWN_POLICY = 0;
+ HOT = 1;
+ WARM = 2;
+ COLD = 3;
+}
+
+enum S3StorageClassProto {
+ UNKNOWN_STORAGE_CLASS = 0;
+ STANDARD = 1;
+ STANDARD_IA = 2;
+ GLACIER = 3;
+ DEEP_ARCHIVE = 4;
+}
+
+message Pipeline {
+ //...
+ repeated StorageTierProto supportedStorageTier = 1;
Review Comment:
Great that the pipeline may support multiple storage tiers. What will happen if volumes change: volumes crash, or DN adds/removes volumes between heartbeats. What should operators do with pipelines when DN storage changes (pipelines close/create)?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
+ Datanode, it will try to use the fallback storage types defined by the SCM container's storage tier. If the storage
+ tier has no defined fallback storage types, or no Datanodes matching the fallback storage types can be found, the
+ operation fails.
+
+## DiskBalancerService
+
+- DiskBalancerService calculates space usage and migrates containers between volumes of the same storage type within a
+ Datanode.
+
+# Configurations
+
+New configuration keys introduced by the storage policy feature:
+
+- `ozone.default.storagepolicy`
+ Default value: WARM. The default storage policy for writing keys when neither the key nor the bucket has a storage
+ policy set.
+
+- `ozone.default.storageTier`
+ Default value: DISK. The default storage tier.
+
+- `ozone.s3.default.storagepolicy`
+ Default value: STANDARD_IA. The default S3 storage class when StorageClass is not specified in the S3 request and
+ the bucket does not have a storage policy set. STANDARD_IA maps to Ozone storage policy WARM.
+
+- `hdds.datanode.default.storagetype`
+ Default value: DISK. The default storage type used when creating a container replica on a Datanode without a
+ specified storage type. This typically applies to write requests from older clients that do not support the storage
+ policy feature.
+
+- `ozone.scm.container.allow.null.storage.tier`
+ Default value: false. Whether to allow selecting containers with a null storage tier during container allocation.
+ This can be used during the upgrade period to allow new keys to be written to existing containers that do not yet
+ have a storage tier set.
+
+For a pure SSD cluster (where all Datanode volumes are configured with SSD storage type), the following configurations
+should be updated accordingly:
+
+- `ozone.default.storagepolicy` should be set to HOT (Hot maps to SSD storage tier).
+- `ozone.default.storageTier` should be set to SSD.
+- `ozone.s3.default.storagepolicy` should be set to STANDARD (STANDARD maps to Hot storage policy).
+- `hdds.datanode.default.storagetype` should be set to SSD.
+
+Otherwise, when writing a key without specifying a storage policy, the system will default to the WARM storage policy,
+which corresponds to the DISK storage tier. Since WARM does not define a fallback tier (its fallback is EMPTY), SCM
+cannot fall back to another tier, and the write will fail because no DISK type volumes are available in the cluster.
+
+Note that this situation only occurs when the volumes are explicitly configured as SSD in the `hdds.datanode.dir`
+configuration (e.g., `[SSD]/mnt/disk/0/ozone`). Even if the underlying physical disks are SSDs, as long as the volume
+configuration does not specify a storage type, the Datanode will treat them as DISK type volumes by default, and no
+configuration changes are needed.
+
+# Protobuf Changes
+
+Major protobuf changes:
+
+hdds.proto
+
+```protobuf
+enum StorageType {
+ DISK_TYPE = 1;
+ SSD_TYPE = 2;
+ ARCHIVE_TYPE = 3;
+ RAM_DISK_TYPE = 4;
+ PROVIDED_TYPE = 5;
+}
+
+enum StorageTierProto {
+ UNKNOWN_TIER = 0;
+ DISK_TIER = 1;
+ SSD_TIER = 2;
+ ARCHIVE_TIER = 3;
+}
+
+enum StoragePolicyProto {
+ UNKNOWN_POLICY = 0;
+ HOT = 1;
+ WARM = 2;
+ COLD = 3;
+}
+
+enum S3StorageClassProto {
+ UNKNOWN_STORAGE_CLASS = 0;
+ STANDARD = 1;
+ STANDARD_IA = 2;
+ GLACIER = 3;
+ DEEP_ARCHIVE = 4;
+}
+
+message Pipeline {
+ //...
+ repeated StorageTierProto supportedStorageTier = 1;
+}
+
+message ContainerInfoProto {
+ //...
+ optional StorageTierProto storageTier = 1;
+}
+```
+
+OmClientProtocol.proto
+
+```protobuf
+message BucketInfo {
+ //...
+ optional hadoop.hdds.StoragePolicyProto storagePolicy = 1;
+ optional bool allowFallbackStoragePolicy = 1;
+}
+
+message KeyInfo {
+ //...
+ optional hdds.StoragePolicyProto storagePolicy = 1;
+}
+```
+
+ScmServerDatanodeHeartbeatProtocol.proto
+
+```protobuf
+message StorageReportProto {
+ //...
+ optional StorageType storageTypeProto = 1 [default = DISK_TYPE];
+}
+
+message ContainerReplicaProto {
+ //...
+ optional StorageType storageType = 1;
+ optional StorageType VolumeStorageType = 1;
+}
+```
+
+# HDFS Compatibility
+
+HDFS storage policy-related commands are not supported.
+
+Specifying a storage policy when creating a file using the HDFS Filesystem interface is not supported, as the Hadoop
+FileSystem (org.apache.hadoop.fs.FileSystem) does not support passing the storage policy through the create method.
+
+# Backward Compatibility
+
+This section describes the backward compatibility for data created before the storage policy feature is introduced.
Review Comment:
What is the upgrade scenario? Does it support the rolling upgrade?
My understanding is that the existing cluster should be shut down and the pipelines closed. Then run the upgrade, right?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
Review Comment:
Sounds great. Could you clarify how ReplicationManagers will send that command, and what command?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
+ - ReplicationManager will send a command to update the storage type of the container replica on the Datanode.
+ - ReplicationManager will first handle MIS_STORAGE_TYPE_WITH_CONTAINER and then handle
+ MIS_STORAGE_TYPE_WITH_VOLUME.
+- ReplicationManager uses the SCM container's storage tier as the source of truth to check whether the container
+ replicas and the volumes where replicas reside match the storage tier.
+ - If the SCM container's storage tier is null, no storage tier checks will be performed. Containers created before
+ the storage policy feature was introduced will have a null storage tier.
+ - To manage existing containers' replicas (e.g., to prevent these container replicas from occupying SSD type
+ volume space), the container storage tier needs to be set. Once set, ReplicationManager will manage the
+ container replicas based on the storage tier. Otherwise, container replicas may reside on volumes of any
+ storage type.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas (e.g., for UNDER_REPLICATED or
+ MIS_REPLICATED containers), it selects matching Datanodes based on the storage type corresponding to the SCM
+ container's storage tier.
+- When ReplicationManager replicates, migrates, or reconstructs container replicas, if it cannot find a matching
+ Datanode, it will try to use the fallback storage types defined by the SCM container's storage tier. If the storage
+ tier has no defined fallback storage types, or no Datanodes matching the fallback storage types can be found, the
+ operation fails.
+
+## DiskBalancerService
+
+- DiskBalancerService calculates space usage and migrates containers between volumes of the same storage type within a
+ Datanode.
+
+# Configurations
+
+New configuration keys introduced by the storage policy feature:
+
+- `ozone.default.storagepolicy`
+ Default value: WARM. The default storage policy for writing keys when neither the key nor the bucket has a storage
+ policy set.
+
+- `ozone.default.storageTier`
+ Default value: DISK. The default storage tier.
+
+- `ozone.s3.default.storagepolicy`
+ Default value: STANDARD_IA. The default S3 storage class when StorageClass is not specified in the S3 request and
+ the bucket does not have a storage policy set. STANDARD_IA maps to Ozone storage policy WARM.
+
+- `hdds.datanode.default.storagetype`
+ Default value: DISK. The default storage type used when creating a container replica on a Datanode without a
+ specified storage type. This typically applies to write requests from older clients that do not support the storage
+ policy feature.
+
+- `ozone.scm.container.allow.null.storage.tier`
+ Default value: false. Whether to allow selecting containers with a null storage tier during container allocation.
+ This can be used during the upgrade period to allow new keys to be written to existing containers that do not yet
+ have a storage tier set.
+
+For a pure SSD cluster (where all Datanode volumes are configured with SSD storage type), the following configurations
+should be updated accordingly:
+
+- `ozone.default.storagepolicy` should be set to HOT (Hot maps to SSD storage tier).
+- `ozone.default.storageTier` should be set to SSD.
+- `ozone.s3.default.storagepolicy` should be set to STANDARD (STANDARD maps to Hot storage policy).
+- `hdds.datanode.default.storagetype` should be set to SSD.
+
+Otherwise, when writing a key without specifying a storage policy, the system will default to the WARM storage policy,
+which corresponds to the DISK storage tier. Since WARM does not define a fallback tier (its fallback is EMPTY), SCM
+cannot fall back to another tier, and the write will fail because no DISK type volumes are available in the cluster.
+
+Note that this situation only occurs when the volumes are explicitly configured as SSD in the `hdds.datanode.dir`
+configuration (e.g., `[SSD]/mnt/disk/0/ozone`). Even if the underlying physical disks are SSDs, as long as the volume
+configuration does not specify a storage type, the Datanode will treat them as DISK type volumes by default, and no
+configuration changes are needed.
+
+# Protobuf Changes
+
+Major protobuf changes:
+
+hdds.proto
+
+```protobuf
+enum StorageType {
+ DISK_TYPE = 1;
+ SSD_TYPE = 2;
+ ARCHIVE_TYPE = 3;
+ RAM_DISK_TYPE = 4;
+ PROVIDED_TYPE = 5;
+}
+
+enum StorageTierProto {
Review Comment:
Seems like one more enum, why don't we use `StorageType` directly? It's simpler.
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
Review Comment:
DN may lose volume between pipeline creation and container creation, or due to a race between heartbeat reporting and allocation. Does it mean it will cause write failures?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
Review Comment:
Great! What is the S3/ozone CLI command for that? What is the corresponding proto change for that?
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
Review Comment:
The document mentions filtering pipelines by storage tier, but it does not mention filtering containers within the pipeline by storage tier. Since the pipeline may handle both SSD and DISK, how do we ensure we write to the container on the correct tier?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r2993398281
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,397 @@
+---
+title: Ozone Storage Policy Support
+summary: Support Ozone storage strategy, and support to write key into the specified type of storage medium.
+date: 2024-07-25
+jira: HDDS-11233
+status: draft
+---
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Terminology
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The types of disks/Container replicas in a Datanode, storage type could include RAM_DISK, SSD, HDD, ARCHIVE, etc.
+- Storage Tier: A set of Container replicas in a cluster that satisfy the storage policy.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode..
+- prefix: The prefix in this article, unless otherwise specified, refers to the prefix of the storage policy type, not the ACL prefix. The prefix of the storage policy type is used to configure the prefix of the storage policy for the specified prefix.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relation of Storage Policy, Storage Type and Storage Tier
+
+- The storage policy is the property of key/bucket/ prefix (Managed by OM);
+- The storage tier is the property of Pipeline and Container (Managed by SCM);
+- The storage type is the property of volume and Container replicas (Managed by DN);
+- Only the storage policy can be modified by the user directly via ozone command;
+
+Example:
+
+For a keyA, its storage policy is Hot, its Container 1 tier is SSD tier, and Container 1 has three replicas, all of which are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so create a bucket with the storage policy set to Hot. Data written by User A to bucket will automatically be distributed across the SSD disks in the cluster.
+- User B needs higher IO performance for the directory/prefix /project/metadata, so set the storage policy for the prefix /project/metadata to Hot. Subsequently, data written to /project/metadata will be automatically distributed across the SSD disks in the cluster.
+- User C has already written key1 to the cluster and requires better IO performance. The storage policy for key1 can be set to Hot, and then a migration can be triggered to move key1 to the SSD disks.
+- Use D use command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class XXX` upload a file the Ozone SSD tier
+
+# Current Status
+
+- Ozone currently has some support for tiered storage such as storage type, and some parts of this article may already be implemented.
+- Currently, in Ozone, when a key is created, the key's Block can appear on any volume of a Datanode. When a key is created, SCM first needs to allocate a Block for the key through Pipelines. The Client then writes the Block to the corresponding Datanode based on the Pipeline information. In this process, the smallest element managed by the SCM Pipeline is the Datanode, and when the Datanode creates a Container, the Container may appear on any volume with enough remaining space. Under the current architecture, Ozone does not support writing data to specific disks
+
+# Goal Requirements Specification
+
+### **Support for Storage Policy Writing and Management**
+
+- **Writing keys**: Allow keys to be written to specified storage tiers based on storage policies.
+- **Policy Management**: Enable setting, unsetting, and inheriting storage policies for keys, prefixes, and buckets. Inherit policies based on the longest matching prefix or bucket if no specific policy is set.
+
+### **Support for Data Migration Across Different Storage Policies**
+
+- **Data Migration**: Support data migration across different storage policies via manual triggers, ensuring data is moved to the appropriate storage tiers.
+
+### **Adaptation of AWS S3 StorageClass**
+
+- **S3 StorageClass Mapping**: Map AWS S3 storage classes to Ozone storage policies, supporting related API operations (PutObject, CopyObject, Multipart Upload, GetObject, HeadObject, ListObjects).
+
+### **Management and Monitoring Tools**
+
+- **Storage Policy Commands**: Provide tools to view storage policies of containers, datanode usage, and pipeline information.
+- **Metrics and Monitoring**: Enable visibility into storage policy compliance, container storage types, and space information across different storage policies.
+
+### **Future Enhancements**
+
+- **Intelligent Storage Policies**: Plan to support automatic data migration based on access frequency, similar to S3 Intelligent-Tiering.
+- **Bucket StorageClass Lifecycle Rules: Support setting storage policies Lifecycle Rules at the bucket level.**
+- **Recon Support**: Enhance Recon to display relevant storage tier information.
+
+# Detailed Requirements Specification
+
+## Storage Policy and Storage Types
+
+### Supported Storage Types
+
+- Specify the Storage Type for each volume through configuration. If no Storage Type is specified, the default value will be DISK.
+- Support Storage Type:SSD / DISK / ARCHIVE / RAM_DISK
+
+### Supported Storage Policies
+
+Support storage policy: Hot , Warm, Cold
+
+### Storage Policies Map To Storage Tiers
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+| --- | --- | --- |
+| Hot | SSD | DISK |
+| Warm | DISK | none |
+| Cold | ARCHIVE | none |
+- **Storage Tier For Write**: The priority storage tier where data is written when storage policy is specified.
+- **Fallback Tier for Write**: If the specified storage policy cannot be satisfied with the priority storage tier, the SCM will attempt to use this fallback tier to meet the policy requirements.
+
+### Storage Tier Map To Storage Type
+
+| Tier | StorageType of Pipeline | One Replication
+Container Replicas Storage Type | Three replication
+Container Replicas Storage Type | EC
+Container Replicas Storage Type |
+| --- | --- | --- | --- | --- |
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+
+### Fallback Storage Type For Container replicas Replication/Migration
+
+| Container Replicas Type | Container Replicas Fallback Storage Type [1] |
+| --- | --- |
+| SSD | DISK |
+| DISK | none |
+| ARCHIVE | none |
+- Container Replicas Fallback Storage Type: During the Container replicas replication or migration process, if the SCM cannot find a suitable volume type that matches the original Container replica's storage type, it will attempt to use this fallback storage tier.
+
+[1] For a Container replicas, it will not know the Storage Policy of the Container’s key or the tier of the SCM Container located, the Container replicas just know its own expected storage type, So column name is “Fallback Storage Type”
+
+## Support for Ozone Storage Policy Writing and Management
+
+### Support storage policy writing
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If no storage policy is specified, the default behavior refers to the "Inheritance of storage policy" section.
+ - If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy (can refers to the "default storage policy" section)
+ - If the priority storage policy is not satisfied, support writing to the fallback tier if the fallbackStrategy is “allow”
+
+### Support fallback strategy configuration
+
+- fallbackStrategy
+ - Allow (default): In this case, the behavior is similar to HDFS, with automatic fallback, and it does not trigger errors or additional alerts;
+ - Prohibit: Prohibit fallback; if a tier that satisfies the storage policy cannot be found, the write operation fails directly.
+
+### Inheritance of storage policies
+
+- If no storage policy is specified (undefined storage policy) when writing a key, the key's storage policy inherits the longest matching prefix. If there is no matching prefix, it inherits the storage policy of the bucket. If the bucket has no effective storage policy [1], the key's storage policy will be the default storage policy .
+- If a key is created with an effective storage policy, the storage policy of the key will not change with the storage policy changing of the bucket or prefix.
+
+[1] Effective storage policy means a non-empty storage policy.
+
+### undefined storage policy
+
+- If the user does not specify any storage policy when creating a key, the user's storage policy is undefined storage policy.
+- Even if the user's key inherits the storage policy of the bucket/prefix, the user's storage policy is still undefined storage policy.
+- Undefined storage policy does not mean the key no storage policy. if the key inherits a storage policy, the key actual storage policy is the inherited storage policy.
+- The undefined storage policy will change as the changing of the prefix/bucket storage policy, including when the key is renamed to a prefix with a different storage policy.
+
+### default storage policy
+
+- If a key neither inheriting any storage policy nor specified a storage policy when writing a key, then the key storage policy will be default storage policy
+- The default storage policy is the storage policy for existing keys before the storage policy feature is launched. That is, all keys have at least the default storage policy, even if the key was created before the storage policy feature was launched.
+- If the user has not configured a default storage policy, the default storage policy should be Warm.
+- The default storage policy can be configured.
+
+## Storage policy management
+
+### key storage policy
+
+- Support setting and unsetting the storage policy for keys.
+ - After unsetting the storage policy, the actual storage policy of the key refers to the "Inheritance of storage policy" section, or is the default storage policy if key do not inherit any storage policy.
+- Support displaying the storage policy in the key list/info results(Include whether the storage policy is the default storage policy).
+
+### Bucket storage policy management
+
+- Support setting and unsetting storage policies for buckets.
+ - After unsetting the storage policy, the storage policy is the default storage policy.
+- Support specifying a storage policy when creating a bucket.
+ - If no storage policy is specified, the storage policy is the default storage policy.
+- Support displaying the storage policy in the bucket list/info results (Include whether the storage policy is the default storage policy).
+
+### Prefix management
+
+- Support creating, deleting, setting, getting, and listing prefixes.
+- The display of the prefix storage policy should display whether the storage policy is the default storage policy
+- A prefix can only have one type of policy.
+- Prefixes do not support unsetting storage policies; deleting a prefix is equivalent to unsetting the storage policy.
+
+### Support for persistent storage of storage policy changes:
+
+- Use ozone admin storagepolicies satisfyStoragePolicy to trigger the migration of corresponding changes, and mark the corresponding storage policy changes as completed.
+
+### FSO type buckets:
+
+- Use prefixes to implement directory-level storage policy management, not directly support setting storage policies for directories.
+- Do not support setting storage policies for directory-type keys.
+
+## Adaptation of AWS S3
+
+### Adaptation of AWS S3 StorageClass
+
+Not all the StorageClass will be support by the Ozone
+
+A possible solution
+
+| AWS S3 StorageClass | Ozone StoragePolicy |
+| --- | --- |
+| STANDARD | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | COLD |
+
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD, which is not easy to convert it to OZONE SSD
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+
+### Adaptation of AWS S3 Related API
+
+refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage policy when migrating the object from the source location to the target location. If no new storage policy is specified, inherit the storage policy of the source object.
+- Multipart Upload
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass parameter of CreateMultipartUpload in UploadPart.
+- GetObject operation:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Support the HeadObject request to return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+- Bucket StorageClass Lifecycle Rules:
+ - Support setting storage policies Lifecycle Rules at the bucket level and automatically managing the storage policy conversion of objects through policies and lifecycle rules. For example, automatically transfer objects from SSD to HDD or from HDD to NVMe SSD based on the object's age or access frequency.
+- ~~RestoreObject operation: [Not Supported]~~
+
+## Support for Storage Policy Management Commands/Metrics
+
+Lists commands that need to be added/adapted to storage policies, but may not be all commands
+
+- Storage policy management
+ - `ozone admin storagepolicies list`, lists all supported policies in the current cluster.
+ - `ozone admin storagepolicies get ${key}`, get the key storage policy (include the inherited storage policy)
+ - `ozone admin storagepolicies check ${key}`, check whether the key storage policy is satisfied
+ - `ozone admin storagepolicies satisfyStoragePolicy -bucket / -prefix / -key`, triggers migrations to satisfy the corresponding data's storage policy.
Review Comment:
Regarding the Job Worker architecture, I believe integrating it directly into Ozone is a better approach than having it as a standalone component.
The workflow could involve the Ozone Manager (OM) detecting migration needs while the Storage Container Manager (SCM) orchestrates the actual data movement. The primary advantage of integration is the ability to leverage the most comprehensive and real-time cluster information.
We could approach this within an OM listener to reduce the loading from the OM Leader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3850912514
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
+cluster so that they have the specified storage tier attribute.
+
+- `ozone admin container setstoragetier --storage-tier=<storageTierStr>`
+ Allowed values: SSD, DISK, ARCHIVE, or null to unset.
+
+## Storage Policy Display
+
+Display the bucket storage policy attribute:
+
+- `ozone sh bucket info`
+
+Display the key storage policy attribute:
+
+- `ozone sh key info`
+
+Display the container storage tier, the storage type of its replicas, and the storage type of the volumes where the
+replicas reside:
+
+- `ozone admin container info`
+
+List containers filtered by storage tier:
+
+- `ozone admin container list --storage-tier=<storageTierStr>`
+
+Display the storage tiers supported by pipelines associated with Datanodes:
+
+- `ozone admin datanode list`
+
+Display Datanode usage information by storage type:
+
+- `ozone admin datanode usageinfo`
+
+Display cluster-wide space usage information by storage type:
+
+- `ozone admin storagepolicy usageinfo`
+
+Check whether a key's storage policy is satisfied:
+
+- `ozone admin storagepolicy check`
+
+## Container Balancer
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Container Balancer supports balancing based on the space usage of each storage type. Each storage type is calculated
+ independently. Container replicas are migrated to volumes of the matching storage type based on the container
+ replica's storage type.
+- Container Balancer does not perform fallback during migration. It will not migrate a container replica to a volume
+ of a different storage type.
+
+## ReplicationManager
+
+- Datanodes report the status information of all containers to SCM through ContainerReplicaProto, including the
+ storage type of the container replica itself and the storage type of the volume where it resides.
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- ReplicationManager introduces two new container health states: MIS_STORAGE_TYPE_WITH_VOLUME and
+ MIS_STORAGE_TYPE_WITH_CONTAINER.
+ - MIS_STORAGE_TYPE_WITH_VOLUME: Container replica whose storage type mismatches its volume's storage type.
+ - ReplicationManager will select a matching volume and migrate the container replica for this health state.
+ - MIS_STORAGE_TYPE_WITH_CONTAINER: Container replica whose storage type mismatches the container's storage tier.
Review Comment:
- An old client creates a container replica on the datanode without carrying the `storageType`, leaving the replica's storage type null.
- ReplicationManager repairs or migrates a replica using a fallback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3772977410
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
Review Comment:
is any state container allowed here to set the storage tier? Is there any action triggered later to move the container to the corresponding tier?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3773401970
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
+For example, if all Datanodes in a pipeline have both SSD and DISK type volumes, the pipeline's supported storage tier
+attributes will include both SSD and DISK.
+
+## Datanode Volume Storage Type
+
+- Referencing HDFS [1], use configuration to define the storage type of each volume. If no storage type information is
+ configured for a volume, the storage type will be DISK.
+ - For example:
+ ```xml
+ <property>
+ <name>hdds.datanode.dir</name>
+ <value>[SSD]/mnt/disk/0/ozone,[DISK]/mnt/disk/1/ozone</value>
+ </property>
+ ```
+ Volume /mnt/disk/0/ozone will be SSD storage type, and Volume /mnt/disk/1/ozone will be DISK storage type.
+
+[1] Refer to https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html section
+Configuration.
+
+## Writing Keys with Storage Policy
+
+### Ozone Filesystem
+
+- Support specifying a storage policy when writing a key.
+ - If a storage policy is specified when writing a key, the key storage policy is the specified storage policy.
+ - If a storage policy is not specified when writing a key, the storage policy of the key will depend on the bucket.
+ If the bucket does not have a storage policy set, the default storage policy is determined by the configuration
+ `ozone.default.storagepolicy`, the default value is WARM.
+
+- Fallback strategy:
+ - When writing a key, if data cannot be written to the specified storage tier, it can be written to the fallback
+ storage tier, provided the fallback storage tier is not EMPTY and fallback is allowed.
+ - Fallback control: Fallback can be controlled at the bucket granularity. If allowed, the key will attempt to write
+ to the fallback storage tier. The default is to allow fallback.
+
+### S3
+
+- If StorageClass is specified in the request, the storage policy is derived from the StorageClass in the request.
+- If StorageClass is not specified in the request, the storage policy of the bucket is used.
+- If the storage policy of the bucket is null, the storage policy is determined by the configuration
+ `ozone.s3.default.storagepolicy`, the default value is STANDARD_IA, which maps to Ozone storage policy WARM.
+
+### Supported APIs
+
+#### Ozone Filesystem
+
+- createKey
+- createStreamKey
+- createFile
+- createStreamFile
+- initiateMultipartUpload
+
+#### S3 Request
+
+Refer to
+[Using Amazon S3 storage classes - Amazon Simple Storage Service](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html)
+
+- PutObject:
+ - Support specifying the StorageClass parameter in the PutObject request to determine the storage policy for the
+ object.
+- CopyObject:
+ - Support specifying a new storage policy (StorageClass) in the CopyObject request and applying the new storage
+ policy when copying the object from the source location to the target location. If no new storage policy is
+ specified, inherit the storage policy of the source object.
+- Multipart Upload:
+ - Support specifying the StorageClass parameter in the CreateMultipartUpload request and following the StorageClass
+ parameter of CreateMultipartUpload in UploadPart.
+- GetObject:
+ - Return the current storage policy of the object in the GetObject response.
+- HeadObject:
+ - Return the metadata of the object, including its storage policy (StorageClass).
+- ListObjects:
+ - Include the storage policy (StorageClass) information of each object in the ListObjects response.
+
+### Write Key Process
+
+#### Normal Write (No Fallback)
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM allocates a container and pipeline with storage tier SSD according to storage policy Hot.
+- Client determines the storage type of the chunk to be SSD according to the storage tier SSD.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+#### Fallback Write
+
+
+
+- Client sends CreateKey request to OM with storage policy Hot.
+- OM requests SCM to allocate a container with storage policy Hot.
+- SCM attempts to allocate a container and pipeline with storage tier SSD according to storage policy Hot, but finds
+ that there are no available containers or pipelines with SSD tier. SCM then selects the fallback tier DISK.
+- Client determines the storage type of the chunk to be DISK according to the fallback storage tier DISK.
+- DN creates the container or writes the chunk on the specified storage medium according to the storage type.
+
+### Datanode Container Replica Creation
+
+- When the Ozone client writes data to a container replica on a Datanode, the storage type is included in the request.
+- When the Datanode receives a write request for a container replica and the container does not exist, the Datanode
+ creates the container by selecting an appropriate volume (VolumeChoosingPolicy) based on the storage type in the
+ request.
+- If the storage type is not specified when creating a container replica on the Datanode, the container replica will be
+ created on a volume of the storage type specified by the configuration `hdds.datanode.default.storagetype` (default
+ is DISK). This typically only occurs when a client from an older version that does not support the storage policy
+ feature writes data to the Datanode.
+- The Datanode does not perform fallback when creating container replicas. If a volume of the storage type specified in
+ the client's write request cannot be found, an exception will be thrown. Since SCM selects appropriate Datanodes to
+ create pipelines and allocate containers based on Datanode reports, this situation should not occur during normal
+ operation.
+
+### SCM Pipeline Creation and Selection
+
+- Datanodes report their storage space information to SCM through StorageReportProto, including the space usage and
+ storage type of all volumes (the default storage type for a Datanode volume is DISK).
+- SCM calculates the supported volume types and capacities for each Datanode based on StorageReportProto.
+- When creating a container, SCM selects appropriate Datanodes to create a pipeline based on the storage tier
+ corresponding to the storage policy in the request.
+ - For example, the Hot storage policy corresponds to the SSD storage tier. For a 3-replica SSD storage tier, the
+ pipeline will consist of 3 Datanodes that have SSD type volumes.
+ - For the Hot storage policy with EC 6+3 replication, the pipeline will consist of 9 Datanodes that have SSD
+ type volumes.
+- If SCM cannot find enough suitable Datanodes to create a pipeline, it will try to create a pipeline using the
+ fallback tier of the storage policy. If the fallback tier is EMPTY, or the fallback tier also cannot find enough
+ suitable Datanodes, the creation fails.
+- SCM does not mix storage types within a storage tier. An SSD storage tier pipeline will only consist of Datanodes
+ with SSD type volumes, and will not include Datanodes with only DISK type volumes.
+- The storage tier is an attribute of the pipeline, indicating the storage tiers supported by that pipeline. A pipeline
+ can support multiple storage tiers. For example, if Datanodes all have both SSD and DISK type volumes, the pipeline
+ created on these Datanodes will have supported storage tier attributes of both SSD and DISK.
+- When allocating a container through an existing pipeline, SCM filters matching pipelines based on their supported
+ storage tiers.
+
+
+- BackgroundPipelineCreator will attempt to create pipelines for all storage tiers. If the Datanodes in the cluster
+ can support a certain storage tier, BackgroundPipelineCreator will automatically create pipelines for that tier.
+- The pipeline count limit is calculated independently for each storage tier.
+
+## Storage Policy Update
+
+Setting the bucket storage policy is supported. Existing buckets can be updated with a new storage policy.
+
+- `ozone sh bucket update --storagepolicy <storagePolicyStr>`
+ Allowed values: HOT, WARM, COLD, or null to unset.
+
+Setting the SCM container storage tier attribute is supported. This is mainly used to update existing containers in the
Review Comment:
After finish the whole review, I would suggest we don't provide this container storage tier CLI to user, since we already support the "no storage policy" key and "no storage tier" container already welly. In what case does the user need to manually change/set container storage tier?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11233. Ozone Storage Policy Support. [ozone]
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #6989:
URL: https://github.com/apache/ozone/pull/6989#discussion_r3772954987
##########
hadoop-hdds/docs/content/design/storage-policy.md:
##########
@@ -0,0 +1,607 @@
+---
+title: Ozone Storage Policy Support
+summary: Support storage policy in Ozone to write key data into specified types of storage media.
+date: 2026-03-23
+jira: HDDS-11233
+status: draft
+---
+
+<!--
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+# Terminology
+
+## Definitions
+
+- Storage Policy: Defines where key data replicas should be stored in specific storage tiers.
+- Storage Type: The type of each Datanode volume or container replica. Each Datanode volume can be configured with a
+ storage type, including SSD, DISK, and ARCHIVE.
+- Storage Tier: A specific storage tier is composed of all replicas of a container based on their storage type. For
+ example, a 3-replica SSD tier consists of 3 replicas of SSD type.
+- Volume: In this document, unless otherwise specified, a volume refers to the volume of a Datanode.
+- Key: In this document, a key refers to an object in Ozone, including entries in both the KeyTable and FileTable.
+
+## Storage Policy vs Storage Type vs Storage Tier
+
+
+
+The relationship between Storage Policy, Storage Type, and Storage Tier:
+
+- The storage policy is the property of key/bucket (managed by OM).
+- The storage tier is the property of Pipeline and Container (managed by SCM).
+- The storage type is the property of volume and container replica (managed by DN).
+- Only the storage policy can be modified by the user directly via the ozone command.
+
+Example:
+
+For a keyA, its storage policy is Hot, Its Container tier is SSD tier, the Container has three replicas, all of which
+are of the SSD storage type.
+
+# User Scenarios
+
+- User A needs a bucket that supports high-performance IO, so they create a bucket with the storage policy set to Hot.
+ Data written by User A to the bucket will automatically be distributed across SSD disks in the cluster.
+- User B needs higher IO performance for a specific key. They write a key with the storage policy set to Hot. The
+ key's data will be distributed across SSD disks in the cluster.
+- User C uses the command `aws s3 cp myfile.txt s3://my-bucket/myfile.txt --storage-class STANDARD` to upload a file
+ to the Ozone SSD tier. The key's data will be distributed across SSD disks in the cluster.
+
+# Goals
+
+- Storage Policy: Introduce storage policy and related concepts. Define multiple storage policies and support S3
+ storage class.
+- Storage Policy Writing: Allow writing keys/files to specified storage tiers based on storage policy. Support S3,
+ API, and shell command interfaces.
+- Storage Policy Update: Enable setting and unsetting storage policies for buckets, and setting storage tiers for
+ containers.
+- Storage Policy Display: Support displaying the storage policy attribute of buckets and keys. Support displaying the
+ storage tier of SCM containers and pipelines. Support displaying Datanode storage type usage information. Support
+ checking whether the key storage policy is satisfied.
+- Container Balancer: Support migrating container replicas between Datanodes to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+- ReplicationManager: Support managing the storage type of container replicas to ensure that container replicas on
+ Datanodes reside on the correct volumes. Ensure that the storage types of container replicas forming a storage
+ tier are correct. For example, a 3-replica SSD storage tier container in SCM should consist of 3 SSD type container
+ replicas, and each container replica should reside on an SSD type volume.
+- DiskBalancerService: Support migrating container replicas within a Datanode to volumes of the matching storage type.
+ For example, SSD type container replicas will be migrated to SSD type volumes, and will not be migrated to DISK
+ type volumes.
+
+# Design
+
+## Supported Storage Policies
+
+- Supported storage policies: Hot / Warm / Cold
+- Supported storage tiers: SSD / DISK / ARCHIVE / EMPTY
+- Supported storage types: SSD / DISK / ARCHIVE
+- Supported bucket layouts: FILE_SYSTEM_OPTIMIZED, OBJECT_STORE, LEGACY
+- S3 storage classes: STANDARD / STANDARD_IA / GLACIER
+
+### Storage Policy Map to Storage Tier
+
+| Storage Policy | Storage Tier for Write | Fallback Tier for Write |
+|----------------|------------------------|-------------------------|
+| Hot | SSD | DISK |
+| Warm | DISK | EMPTY |
+| Cold | ARCHIVE | EMPTY |
+
+- Storage Tier for Write: The primary storage tier where data is written when a storage policy is specified.
+- Fallback Tier for Write: If the specified storage policy cannot be satisfied with the primary storage tier, SCM
+ will attempt to use this fallback tier to meet the policy requirements. EMPTY means no fallback is available.
+
+### Storage Tier Map to Storage Type
+
+| Tier | Storage Type of Pipeline | One Replica Container Storage Type | Three Replica Container Storage Type | EC Container Replicas Storage Type |
+|---------|--------------------------|-------------------------------------|--------------------------------------|-------------------------------------|
+| SSD | SSD | SSD | 3 SSD | n SSD |
+| DISK | DISK | DISK | 3 DISK | n DISK |
+| ARCHIVE | ARCHIVE | ARCHIVE | 3 ARCHIVE | n ARCHIVE |
+| EMPTY | - | - | - | - |
+
+### Fallback Storage Type for Container Replica Replication/Migration
+
+| Container Replica Storage Type | Fallback Storage Types (ordered) [1] |
+|--------------------------------|--------------------------------------|
+| SSD | DISK, ARCHIVE |
+| DISK | ARCHIVE |
+| ARCHIVE | none |
+
+- Fallback Storage Type: During the container replica replication or migration process, if SCM cannot find a suitable
+ volume type that matches the original container replica's storage type, it will attempt to use the fallback storage
+ types in order.
+
+[1] A container replica does not know the storage policy of the key or the storage tier of the SCM container it belongs
+to. The container replica only knows its own expected storage type, which is why the column name is "Fallback Storage
+Types" rather than "Fallback Storage Tier".
+
+### AWS S3 StorageClass
+
+| AWS S3 StorageClass | Ozone Storage Policy |
+|---------------------|----------------------|
+| STANDARD [1] | Hot |
+| STANDARD_IA | Warm |
+| GLACIER | Cold |
+| DEEP_ARCHIVE | Warm |
+
+> AWS StorageClass Valid Values: STANDARD | REDUCED_REDUNDANCY | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING |
+> GLACIER | DEEP_ARCHIVE | OUTPOSTS | GLACIER_IR | SNOW | EXPRESS_ONEZONE
+> According to AWS S3 documentation, STANDARD is the highest performance S3 StorageClass, but its name is STANDARD,
+> which is not straightforward to map to the Ozone SSD tier.
+
+[1] The field names here reuse the AWS S3 field names, but the actual semantics differ from AWS S3. For example, in
+Ozone, STANDARD represents the Hot storage policy, while in AWS S3, STANDARD has different semantics.
+
+## Component Changes
+
+### Datanode Container Replica
+
+A storage type field is added to container replicas on Datanodes, which is persisted in the container's metadata YAML
+file.
+
+### Bucket, Key
+
+A storage policy attribute is added to buckets and keys on OM.
+
+### SCM Container, Pipeline
+
+A storage tier attribute is added to containers and pipelines on SCM. A pipeline can support multiple storage tiers.
Review Comment:
> A pipeline can support multiple storage tiers.
Is it because that write pipeline is heavy, so a pipeline support have multiple storage tiers, to reduce the overall pipeline count related? There is one thing that if some write error happens on HDD, datanode will close the whole pipeline, which will make the data write requests for both HDD and SSD retry. Retry for HDD might be acceptable, while SSD writes are most latency sensitive, so there could be latency spikes in this case. Have we evaluated the one pipeline single storage tier solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org