You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Íñigo Goiri (JIRA)" <ji...@apache.org> on 2018/10/17 17:05:00 UTC
[jira] [Resolved] (HDFS-13378) Add the option to set the du reserve
by percent on datanode
[ https://issues.apache.org/jira/browse/HDFS-13378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Íñigo Goiri resolved HDFS-13378.
--------------------------------
Resolution: Duplicate
> Add the option to set the du reserve by percent on datanode
> -----------------------------------------------------------
>
> Key: HDFS-13378
> URL: https://issues.apache.org/jira/browse/HDFS-13378
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 3.0.1
> Reporter: Gang Xie
> Priority: Major
> Attachments: HDFS-13378_v1.diff
>
>
> Recently, we could event we reserved some space(dfs.datanode.du.reserved), this disk still gets full, in turn, the performance is downgraded. After some test, we found that we need consider the space used by ext4 (we use ext4 as local fs). I.E, if the datanode reports there is 8T space available, you could not write 8T data actually. About %4-%5 space is used for the ext4 metadata. The following is the metrics of ext4 meta usage in disks with different capacity (GB, reserved block for root is set to 1%, 120MB blocks):
> |*FileSystem Capacity*|*Ext4 Data*|*Ex4 metadata*|*Metadata Ratio*|
> |3666|3629|37|0.0101|
> |7392|7318|74|0.0100|
> |733|725|8|0.0109|
>
> So, it seems that the ext4 most likely has the same meta data ratio with the same about data and dir tree structure.
>
> On the other hand, in our data center, there are several disk types with different capacity. It's inefficient to set a absolute value for each datanode according to the disk capacity. To us, it makes sense to set the reservation with a ratio since we always use ext4 as the local fs. I think it's the same case for most of other data center.
>
> Our idea is to leave the default behavior unchanged. User could still use dfs.datanode.du.reserved to set the reservation, but switch to use a ratio if some other specified param is set.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org