You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2009/12/24 19:22:29 UTC
[jira] Created: (HDFS-853) The HDFS webUI should show a metric that
summarizes whether the cluster is balanced regarding disk space usage
The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
--------------------------------------------------------------------------------------------------------------
Key: HDFS-853
URL: https://issues.apache.org/jira/browse/HDFS-853
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Reporter: dhruba borthakur
It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-853) The HDFS webUI should show a metric that
summarizes whether the cluster is balanced regarding disk space usage
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HDFS-853:
----------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Fix Version/s: 0.22.0
Resolution: Fixed
I just committed this. Thanks Dmytro!
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-853.patch, Screen shot 2010-04-27 at 5.52.05 PM.png
>
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-853) The HDFS webUI should show a metric
that summarizes whether the cluster is balanced regarding disk space usage
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865031#action_12865031 ]
dhruba borthakur commented on HDFS-853:
---------------------------------------
Konstantin: does this patch address your concern of whether it will handle heterogeneous clusters?
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: HDFS-853.patch, Screen shot 2010-04-27 at 5.52.05 PM.png
>
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-853) The HDFS webUI should show a metric
that summarizes whether the cluster is balanced regarding disk space usage
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur reassigned HDFS-853:
-------------------------------------
Assignee: Dmytro Molkov
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-853) The HDFS webUI should show a metric that
summarizes whether the cluster is balanced regarding disk space usage
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-853:
-------------------------------
Attachment: Screen shot 2010-04-27 at 5.52.05 PM.png
HDFS-853.patch
Attaching a patch and a screenshot of what it looks like. Please have a look at it.
No unittest included since it is a UI change rather than anything else, but if you can think of a way to test it - let me know.
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: HDFS-853.patch, Screen shot 2010-04-27 at 5.52.05 PM.png
>
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-853) The HDFS webUI should show a metric
that summarizes whether the cluster is balanced regarding disk space usage
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796795#action_12796795 ]
Konstantin Shvachko commented on HDFS-853:
------------------------------------------
May be we should use the mean and standard deviation of _utilization_ rather than directly disk space. This would work for heterogeneous clusters as well. By utilization I mean the percentage of disk space used for blocks on a data-node. We should also make sure this is consistent with the Balancer: balancing should improve the metrics.
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-853) The HDFS webUI should show a metric
that summarizes whether the cluster is balanced regarding disk space usage
Posted by "Andrew Ryan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796772#action_12796772 ]
Andrew Ryan commented on HDFS-853:
----------------------------------
We're currently graphing both mean and standard deviation of datanodes from that mean, using a script that parses the output of 'dfsadmin -report'. Our DFS cluster nodes all have the same amount of disk space, so you'd expect mean of individual datanodes to be the same as % DFS full, but it's not quite the same. Haven't yet looked into why this is so.
To directly answer Konstantin's question, the one line we're using is standard deviation.
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-853) The HDFS webUI should show a metric
that summarizes whether the cluster is balanced regarding disk space usage
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794488#action_12794488 ]
Konstantin Shvachko commented on HDFS-853:
------------------------------------------
How do you propose to measure the variance? Calculate the average utilization and then print in a separate column the difference in %% from that average for each node?
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-853) The HDFS webUI should show a metric
that summarizes whether the cluster is balanced regarding disk space usage
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865876#action_12865876 ]
dhruba borthakur commented on HDFS-853:
---------------------------------------
+1 code looks good.
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: HDFS-853.patch, Screen shot 2010-04-27 at 5.52.05 PM.png
>
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-853) The HDFS webUI should show a metric
that summarizes whether the cluster is balanced regarding disk space usage
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796455#action_12796455 ]
Konstantin Shvachko commented on HDFS-853:
------------------------------------------
OK, so you want one line that reflects the cluster-wide balance rather than a deviation from the average for each data-node.
How do you measure the balance? Do you have any particular measurement in mind?
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-853) The HDFS webUI should show a metric
that summarizes whether the cluster is balanced regarding disk space usage
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861665#action_12861665 ]
Hadoop QA commented on HDFS-853:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12443028/HDFS-853.patch
against trunk revision 937914.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/330/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/330/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/330/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/330/console
This message is automatically generated.
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: HDFS-853.patch, Screen shot 2010-04-27 at 5.52.05 PM.png
>
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-853) The HDFS webUI should show a metric that
summarizes whether the cluster is balanced regarding disk space usage
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-853:
-------------------------------
Status: Patch Available (was: Open)
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: HDFS-853.patch, Screen shot 2010-04-27 at 5.52.05 PM.png
>
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-853) The HDFS webUI should show a metric
that summarizes whether the cluster is balanced regarding disk space usage
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794473#action_12794473 ]
dhruba borthakur commented on HDFS-853:
---------------------------------------
We can calculate the variance of space-usage by parsing "dfsadmin -report" but it will be good to have it in the webUI. It will also be good to expose it via a Hadoop Metric.
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-853) The HDFS webUI should show a metric
that summarizes whether the cluster is balanced regarding disk space usage
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794584#action_12794584 ]
dhruba borthakur commented on HDFS-853:
---------------------------------------
I would like a cluster-wide measure that describes how much the cluster is balanced. The Cluster Summary prints total capacity, used disk space, etc. and it would be good to include another line that reflects how-balanced the cluster is.
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-853) The HDFS webUI should show a metric
that summarizes whether the cluster is balanced regarding disk space usage
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865872#action_12865872 ]
Dmytro Molkov commented on HDFS-853:
------------------------------------
This patch should address any clusters nicely since what we are operating with is the load percentage, not absolute values.
> The HDFS webUI should show a metric that summarizes whether the cluster is balanced regarding disk space usage
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-853
> URL: https://issues.apache.org/jira/browse/HDFS-853
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: HDFS-853.patch, Screen shot 2010-04-27 at 5.52.05 PM.png
>
>
> It is desirable to know how much the datanodes vary form one another in terms of space utilization to get a sense of how well a HDFS cluster is balanced.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.