[jira] [Commented] (HDFS-15547) Dynamic disk-level tiering

["Leon Gao (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-15547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186860#comment-17186860 ] 

Leon Gao commented on HDFS-15547:
---------------------------------

[~sureshms] [~weichiu] [~sunchao] This is the new use case of archival storage we discussed. I will upload some initial changes soon.

[~jeffreyz] [~ekanth] FYI

> Dynamic disk-level tiering
> --------------------------
>
>                 Key: HDFS-15547
>                 URL: https://issues.apache.org/jira/browse/HDFS-15547
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode
>            Reporter: Leon Gao
>            Assignee: Leon Gao
>            Priority: Major
>         Attachments: Proposal - Dynamic disk-level tiering.pdf
>
>
> This is a proposal for a new use case based on archival storage, to allow configuring DISK and ARCHIVE storage types on the same device (filesystem) to balance disk IO for disks with different density.
> The proposal is to mainly solve two problems:
> 1) The disk IO of ARCHIVE disks is underutilized. This is normal in many use cases where the data hotness is highly skewed.
> 2) Over the years, as better/cheaper hard drives showing on the market, a large production environment can have mixed disk densities. For example, in our prod environment, we have 2TB, 4TB, 8TB, and 16TB disks. When putting all different HDDs into the cluster, we should be able to utilize disk capacity and disk IO efficiently for all of them.
> When moving blocks from DISK to ARCHIVE, we can prefer the same disk and simply rename the files instead of copying.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org