You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2006/05/24 07:22:30 UTC
[jira] Created: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
HTTP Browsing interface for DFS Health/Status
---------------------------------------------
Key: HADOOP-250
URL: http://issues.apache.org/jira/browse/HADOOP-250
Project: Hadoop
Type: New Feature
Components: dfs
Versions: 0.3
Reporter: Devaraj Das
A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=comments#action_12413164 ]
Doug Cutting commented on HADOOP-250:
-------------------------------------
This looks great. However the method FSNameSystem.getAddress duplicates the logic of DataNode.createSocketAddr. We should have a single static method for this, and it should probably take a Configuration parameter. At a minimum, we should call that method rather than copying it.
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3
> Reporter: Devaraj Das
> Attachments: dfshealth.pdf, http_browsing_dfs.patch
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=comments#action_12413068 ]
Owen O'Malley commented on HADOOP-250:
--------------------------------------
You should be able to use the StatusHttpServer class and add the necessary jsp files for the name node. That will also give you access to the log directories via the web server. It would also make sense to include the free/used disk space on each data node.
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3
> Reporter: Devaraj Das
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=all ]
Sameer Paranjpye updated HADOOP-250:
------------------------------------
Fix Version: 0.3
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3
> Reporter: Devaraj Das
> Fix For: 0.3
> Attachments: dfshealth.pdf, http_browsing_dfs.patch, http_browsing_dfs.patch
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=all ]
Devaraj Das updated HADOOP-250:
-------------------------------
Attachment: dfshealth.pdf
The screen shot of the webpage
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3
> Reporter: Devaraj Das
> Attachments: dfshealth.pdf, http_browsing_dfs.patch
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=all ]
Devaraj Das updated HADOOP-250:
-------------------------------
Attachment: http_browsing_dfs.patch
Attaching a patch which implements the DFS HTTP browsing interface. The default port where the webserver (on the namenode) runs on is 50070. Also attached a screenshot of the webpage.
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3
> Reporter: Devaraj Das
> Attachments: http_browsing_dfs.patch
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=comments#action_12416065 ]
Doug Cutting commented on HADOOP-250:
-------------------------------------
Unit tests are now failing for me with this patch. Can you please look into this? Thanks!
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3.0
> Reporter: Devaraj Das
> Fix For: 0.4.0
> Attachments: dfshealth.pdf, http_browsing_dfs.new.patch, http_browsing_dfs.patch, http_browsing_dfs.patch
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=all ]
Devaraj Das updated HADOOP-250:
-------------------------------
Attachment: http_browsing_dfs.patch
Thanks Doug for the comments. Attached is the revised patch where I call DataNode.createSocketAddr directly.
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3
> Reporter: Devaraj Das
> Attachments: dfshealth.pdf, http_browsing_dfs.patch, http_browsing_dfs.patch
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=comments#action_12414844 ]
Doug Cutting commented on HADOOP-250:
-------------------------------------
I'd like to commit this, but the changes this requires to FSNamesystem.java now conflict with other changes to that file. I can try to sort this out, but if someone else gets a chance first that would be great. Thanks.
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3.0
> Reporter: Devaraj Das
> Fix For: 0.3.0
> Attachments: dfshealth.pdf, http_browsing_dfs.patch, http_browsing_dfs.patch
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=comments#action_12416326 ]
Devaraj Das commented on HADOOP-250:
------------------------------------
Starting an HTTP server within the namenode may be the cause for a slightly longer namenode startup time.
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3.0
> Reporter: Devaraj Das
> Fix For: 0.4.0
> Attachments: dfshealth.pdf, http_browsing_dfs.new.patch, http_browsing_dfs.patch, http_browsing_dfs.patch, http_browsing_dfs.patch
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=all ]
Doug Cutting updated HADOOP-250:
--------------------------------
Fix Version: 0.4
(was: 0.3.0)
Sorry. I am unable to get these into 0.3.1, they'll have to wait for 0.4.
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3.0
> Reporter: Devaraj Das
> Fix For: 0.4
> Attachments: dfshealth.pdf, http_browsing_dfs.patch, http_browsing_dfs.patch
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=all ]
Devaraj Das updated HADOOP-250:
-------------------------------
Attachment: http_browsing_dfs.patch
The problems here were mostly setup related (admit I should have run the unit tests before submitting the patch!).
Since I start a StatusHttpServer in FSNameSystem class, the "build" directory should be included in the classpath (so webapps resource can be found) and hadoop.log.dir should be set, otherwise, StatusHttpServer cannot be constructed properly [updated patch patches build.xml file with these settings].
Also, the stop method of StatusHttpServer should be called every time the namenode is stopped, otherwise a subsequent test trying to create a NameNode object may fail since StatusHttpServer fails to open the port.
Another thing that causes some failures to happen (once in a while) is the sleep/timeout values in MiniDFSCluster.java. A NameNode is started and after sleeping for 1 sec, a datanode is started. But if the namenode hasn't fully come up by then, then RPCs will fail (the ipc.client.timeout of 1 sec seems to be too small). So I changed all 1000 msec to 2000 msec in MiniDFSCluster.java and the tests seem to be passing consistently after that. Updated patch is attached.
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3.0
> Reporter: Devaraj Das
> Fix For: 0.4.0
> Attachments: dfshealth.pdf, http_browsing_dfs.new.patch, http_browsing_dfs.patch, http_browsing_dfs.patch, http_browsing_dfs.patch
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=all ]
Devaraj Das updated HADOOP-250:
-------------------------------
Attachment: http_browsing_dfs.new.patch
Doug, created this patch again against the current trunk.
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3.0
> Reporter: Devaraj Das
> Fix For: 0.4.0
> Attachments: dfshealth.pdf, http_browsing_dfs.new.patch, http_browsing_dfs.patch, http_browsing_dfs.patch
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-250) HTTP Browsing interface for DFS
Health/Status
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-250?page=all ]
Doug Cutting resolved HADOOP-250:
---------------------------------
Resolution: Fixed
I just committed this. I made two additional changes. In FSNameSystem.java I moved the initialization of the static variable fsNameSystemObject to before where Jetty is started, since the jsp pages require a non-null value for this and the servlet init was failing with a null pointer exception. I also added an entry to hadoop-default.xml for the new dfs.info.port parameter.
> HTTP Browsing interface for DFS Health/Status
> ---------------------------------------------
>
> Key: HADOOP-250
> URL: http://issues.apache.org/jira/browse/HADOOP-250
> Project: Hadoop
> Type: New Feature
> Components: dfs
> Versions: 0.3.0
> Reporter: Devaraj Das
> Fix For: 0.4.0
> Attachments: dfshealth.pdf, http_browsing_dfs.new.patch, http_browsing_dfs.patch, http_browsing_dfs.patch, http_browsing_dfs.patch
>
> A web interface to view the DFS health/status (name and data nodes) is to be created. User can connect to the webserver on the namenode and a web page will be displayed. The web page will give some details about that namenode (startup time and the total cluster capacity). The web page will contain a table of 'live' and 'dead' datanodes. Each live datanode will be a link to the complete details of the datanode as given by DatanodeInfo (also see DataNodeReport).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira