You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Eli Collins (Created) (JIRA)" <ji...@apache.org> on 2011/12/20 21:45:30 UTC
[jira] [Created] (HDFS-2708) Stats for the total # blocks and
blocks per DN
Stats for the total # blocks and blocks per DN
----------------------------------------------
Key: HDFS-2708
URL: https://issues.apache.org/jira/browse/HDFS-2708
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node, name-node
Reporter: Eli Collins
Priority: Minor
It would be useful for tools to be able to retrieve the total # blocks in the file system (and also display eg via dfsadmin report, this is currently only available via FSNamesystemMetrics, so would add to ClientProtocol#getStats?) and the total number of blocks on each datanode (via DataNodeInfo).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2708) Stats for the # of blocks per DN
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253933#comment-13253933 ]
Hudson commented on HDFS-2708:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2074 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2074/])
HDFS-2708. Stats for the # of blocks per DN. Contributed by Aaron T. Myers. (Revision 1326039)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326039
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
> Stats for the # of blocks per DN
> --------------------------------
>
> Key: HDFS-2708
> URL: https://issues.apache.org/jira/browse/HDFS-2708
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, name-node
> Affects Versions: 2.0.0
> Reporter: Eli Collins
> Assignee: Aaron T. Myers
> Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-2708.patch
>
>
> It would be useful for tools to be able to retrieve the total number of blocks on each datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2708) Stats for the # of blocks per DN
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2708:
---------------------------------
Description: It would be useful for tools to be able to retrieve the total number of blocks on each datanode. (was: It would be useful for tools to be able to retrieve the total # blocks in the file system (and also display eg via dfsadmin report, this is currently only available via FSNamesystemMetrics, so would add to ClientProtocol#getStats?) and the total number of blocks on each datanode (via DataNodeInfo).)
Target Version/s: 2.0.0 (was: 1.1.0, 0.23.1)
Affects Version/s: 2.0.0
Assignee: Aaron T. Myers
Summary: Stats for the # of blocks per DN (was: Stats for the total # blocks and blocks per DN)
Let's separate the original description of this JIRA into multiple distinct issues. In this JIRA, let's just add support for getting the number of blocks on each DN via the FSNamesystem metrics, since that interface already has support for retrieving the total number of blocks in the file system.
If we really do want to add other interfaces to retrieve both of these metrics, let's do that in a separate JIRA.
> Stats for the # of blocks per DN
> --------------------------------
>
> Key: HDFS-2708
> URL: https://issues.apache.org/jira/browse/HDFS-2708
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, name-node
> Affects Versions: 2.0.0
> Reporter: Eli Collins
> Assignee: Aaron T. Myers
> Priority: Minor
>
> It would be useful for tools to be able to retrieve the total number of blocks on each datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2708) Stats for the # of blocks per DN
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253937#comment-13253937 ]
Hudson commented on HDFS-2708:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2088 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2088/])
HDFS-2708. Stats for the # of blocks per DN. Contributed by Aaron T. Myers. (Revision 1326039)
Result = FAILURE
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326039
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
> Stats for the # of blocks per DN
> --------------------------------
>
> Key: HDFS-2708
> URL: https://issues.apache.org/jira/browse/HDFS-2708
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, name-node
> Affects Versions: 2.0.0
> Reporter: Eli Collins
> Assignee: Aaron T. Myers
> Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-2708.patch
>
>
> It would be useful for tools to be able to retrieve the total number of blocks on each datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2708) Stats for the # of blocks per DN
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253903#comment-13253903 ]
Todd Lipcon commented on HDFS-2708:
-----------------------------------
+1 pending jenkins
> Stats for the # of blocks per DN
> --------------------------------
>
> Key: HDFS-2708
> URL: https://issues.apache.org/jira/browse/HDFS-2708
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, name-node
> Affects Versions: 2.0.0
> Reporter: Eli Collins
> Assignee: Aaron T. Myers
> Priority: Minor
> Attachments: HDFS-2708.patch
>
>
> It would be useful for tools to be able to retrieve the total number of blocks on each datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2708) Stats for the # of blocks per DN
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2708:
---------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks a lot for the review, Todd. I've just committed this to trunk and branch-2.
> Stats for the # of blocks per DN
> --------------------------------
>
> Key: HDFS-2708
> URL: https://issues.apache.org/jira/browse/HDFS-2708
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, name-node
> Affects Versions: 2.0.0
> Reporter: Eli Collins
> Assignee: Aaron T. Myers
> Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-2708.patch
>
>
> It would be useful for tools to be able to retrieve the total number of blocks on each datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2708) Stats for the # of blocks per DN
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253923#comment-13253923 ]
Hadoop QA commented on HDFS-2708:
---------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522641/HDFS-2708.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2275//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2275//console
This message is automatically generated.
> Stats for the # of blocks per DN
> --------------------------------
>
> Key: HDFS-2708
> URL: https://issues.apache.org/jira/browse/HDFS-2708
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, name-node
> Affects Versions: 2.0.0
> Reporter: Eli Collins
> Assignee: Aaron T. Myers
> Priority: Minor
> Attachments: HDFS-2708.patch
>
>
> It would be useful for tools to be able to retrieve the total number of blocks on each datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2708) Stats for the # of blocks per DN
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253932#comment-13253932 ]
Hudson commented on HDFS-2708:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2147 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2147/])
HDFS-2708. Stats for the # of blocks per DN. Contributed by Aaron T. Myers. (Revision 1326039)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326039
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
> Stats for the # of blocks per DN
> --------------------------------
>
> Key: HDFS-2708
> URL: https://issues.apache.org/jira/browse/HDFS-2708
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, name-node
> Affects Versions: 2.0.0
> Reporter: Eli Collins
> Assignee: Aaron T. Myers
> Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-2708.patch
>
>
> It would be useful for tools to be able to retrieve the total number of blocks on each datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2708) Stats for the # of blocks per DN
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2708:
---------------------------------
Attachment: HDFS-2708.patch
Here's a patch for trunk which adds a "numBlocks" field to each DN in the "LiveNodes" field of the NameNodeInfo JMX interface.
> Stats for the # of blocks per DN
> --------------------------------
>
> Key: HDFS-2708
> URL: https://issues.apache.org/jira/browse/HDFS-2708
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, name-node
> Affects Versions: 2.0.0
> Reporter: Eli Collins
> Assignee: Aaron T. Myers
> Priority: Minor
> Attachments: HDFS-2708.patch
>
>
> It would be useful for tools to be able to retrieve the total number of blocks on each datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2708) Stats for the # of blocks per DN
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2708:
---------------------------------
Status: Patch Available (was: Open)
> Stats for the # of blocks per DN
> --------------------------------
>
> Key: HDFS-2708
> URL: https://issues.apache.org/jira/browse/HDFS-2708
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, name-node
> Affects Versions: 2.0.0
> Reporter: Eli Collins
> Assignee: Aaron T. Myers
> Priority: Minor
> Attachments: HDFS-2708.patch
>
>
> It would be useful for tools to be able to retrieve the total number of blocks on each datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2708) Stats for the # of blocks per DN
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254118#comment-13254118 ]
Hudson commented on HDFS-2708:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1049 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1049/])
HDFS-2708. Stats for the # of blocks per DN. Contributed by Aaron T. Myers. (Revision 1326039)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326039
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
> Stats for the # of blocks per DN
> --------------------------------
>
> Key: HDFS-2708
> URL: https://issues.apache.org/jira/browse/HDFS-2708
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, name-node
> Affects Versions: 2.0.0
> Reporter: Eli Collins
> Assignee: Aaron T. Myers
> Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-2708.patch
>
>
> It would be useful for tools to be able to retrieve the total number of blocks on each datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2708) Stats for the # of blocks per DN
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254106#comment-13254106 ]
Hudson commented on HDFS-2708:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1014 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1014/])
HDFS-2708. Stats for the # of blocks per DN. Contributed by Aaron T. Myers. (Revision 1326039)
Result = FAILURE
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326039
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
> Stats for the # of blocks per DN
> --------------------------------
>
> Key: HDFS-2708
> URL: https://issues.apache.org/jira/browse/HDFS-2708
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, name-node
> Affects Versions: 2.0.0
> Reporter: Eli Collins
> Assignee: Aaron T. Myers
> Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HDFS-2708.patch
>
>
> It would be useful for tools to be able to retrieve the total number of blocks on each datanode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira