[jira] Commented: (HADOOP-2816) Cluster summary at name node web has confusing report for space utilization

["Suresh Srinivas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628160#action_12628160 ] 

Suresh Srinivas commented on HADOOP-2816:
-----------------------------------------

For reporting the following info needs to be considered:

Total capacity - Capacity of all the data directories
Reserved space - Space reserved for non DFS usage
dfs.datanode.du.pct - When calculating DFS remaining space, only use this percentage of the real available space

Here is how DFS remaining space is calculated:
Available space is Minimum of (Available space on local file system) or (Total capacity - DFS used space - Reserved space)
DFS remaining = (dfs.datanode.du.pct) * Available space

Current proposal does not consider the factor dfs.datanode.du.pct. I am not sure why du.pct is being used. If it is to reduce available disk space for DFS, to consider factors such as disk fragmentation - it is not serving the purpose. Available space keeps on decreasing. The percentage is applied to the shrinking available space. Eventually the DFS ends up using all the available space any way (in theory) and the du.pct will not serve any purpose.

My proposal:
1) Remove du.pct configuration option

or

2) If du.pct is used, it is calculated on Total capacity and not on available space. This helps set aside a percentage of total capacity.

> Cluster summary at name node web has confusing report for space utilization
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-2816
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2816
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.15.3
>            Reporter: Robert Chansler
>            Assignee: Suresh Srinivas
>
> In one example:
> Cluster Summary
> Capacity	:	1.15 PB
> DFS Remaining	:	192 TB
> DFS Used	:	717 TB
> DFS Used%	:	62 %
> Why is Capacity not equal Used plus Remaining?
> (The answer is that there is an estimated reserve for local files.)
> The presentation should be easily understood by the user.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.