You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Leon Gao (Jira)" <ji...@apache.org> on 2020/08/29 00:21:00 UTC
[jira] [Created] (HDFS-15547) Dynamic disk-level tiering
Leon Gao created HDFS-15547:
-------------------------------
Summary: Dynamic disk-level tiering
Key: HDFS-15547
URL: https://issues.apache.org/jira/browse/HDFS-15547
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode
Reporter: Leon Gao
Assignee: Leon Gao
Attachments: Proposal - Dynamic disk-level tiering.pdf
This is a proposal for a new use case based on archival storage, to allow configuring DISK and ARCHIVE storage types on the same device (filesystem) to balance disk IO for disks with different density.
The proposal is to mainly solve two problems:
1) The disk IO of ARCHIVE disks is underutilized. This is normal in many use cases where the data hotness is highly skewed.
2) Over the years, as better/cheaper hard drives showing on the market, a large production environment can have mixed disk densities. For example, in our prod environment, we have 2TB, 4TB, 8TB, and 16TB disks. When putting all different HDDs into the cluster, we should be able to utilize disk capacity and disk IO efficiently for all of them.
When moving blocks from DISK to ARCHIVE, we can prefer the same disk and simply rename the files instead of copying.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org