You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2007/03/19 22:19:32 UTC
[jira] Created: (HADOOP-1133) Tools to analyze and debug namenode
on a production cluster
Tools to analyze and debug namenode on a production cluster
-----------------------------------------------------------
Key: HADOOP-1133
URL: https://issues.apache.org/jira/browse/HADOOP-1133
Project: Hadoop
Issue Type: Improvement
Components: dfs
Reporter: dhruba borthakur
Assigned To: dhruba borthakur
I would like to have a tool that dumps namenode data structures to a log file. This is especially useful in debugging a production cluster where attaching a debugger to the namenode is not a feasible option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1133) Tools to analyze and debug namenode
on a production cluster
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486925 ]
Hadoop QA commented on HADOOP-1133:
-----------------------------------
Integrated in Hadoop-Nightly #48 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/48/)
> Tools to analyze and debug namenode on a production cluster
> -----------------------------------------------------------
>
> Key: HADOOP-1133
> URL: https://issues.apache.org/jira/browse/HADOOP-1133
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Fix For: 0.13.0
>
> Attachments: metasave3.patch
>
>
> I would like to have a tool that dumps namenode data structures to a log file. This is especially useful in debugging a production cluster where attaching a debugger to the namenode is not a feasible option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1133) Tools to analyze and debug namenode
on a production cluster
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-1133:
------------------------------
Resolution: Fixed
Fix Version/s: 0.13.0
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Dhruba!
> Tools to analyze and debug namenode on a production cluster
> -----------------------------------------------------------
>
> Key: HADOOP-1133
> URL: https://issues.apache.org/jira/browse/HADOOP-1133
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Fix For: 0.13.0
>
> Attachments: metasave3.patch
>
>
> I would like to have a tool that dumps namenode data structures to a log file. This is especially useful in debugging a production cluster where attaching a debugger to the namenode is not a feasible option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1133) Tools to analyze and debug namenode
on a production cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483759 ]
dhruba borthakur commented on HADOOP-1133:
------------------------------------------
Hi Raghu, thanks for your comments. I am printing the datanodes in the log because I am looking at this as a support tool and would liekto capture as much data as possible in the log. I can then ask a cluster administrator to run this tool and i will have everything I need to analyze the cluster. I would hate to ask the cluster admin to run multiple commands to generate the info I need.
Regarding the locking question: you are right that we might be a getting a consistent picture of all the data structures that are being dumped. But each structure is locked individually, this should avoid causing any exceptions.
> Tools to analyze and debug namenode on a production cluster
> -----------------------------------------------------------
>
> Key: HADOOP-1133
> URL: https://issues.apache.org/jira/browse/HADOOP-1133
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: metasave.patch
>
>
> I would like to have a tool that dumps namenode data structures to a log file. This is especially useful in debugging a production cluster where attaching a debugger to the namenode is not a feasible option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1133) Tools to analyze and debug namenode
on a production cluster
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482840 ]
Raghu Angadi commented on HADOOP-1133:
--------------------------------------
code looks fine. A few comments:
1) Do we need datanodes? It is available in webui.
2) this will conflict with version upgrade patch ( because of client version change ).
3) locking : can you double check if just one global lock is enough or required? Currently it locks each data structure seperately.. it might be ok.
> Tools to analyze and debug namenode on a production cluster
> -----------------------------------------------------------
>
> Key: HADOOP-1133
> URL: https://issues.apache.org/jira/browse/HADOOP-1133
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: metasave.patch
>
>
> I would like to have a tool that dumps namenode data structures to a log file. This is especially useful in debugging a production cluster where attaching a debugger to the namenode is not a feasible option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1133) Tools to analyze and debug namenode
on a production cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1133:
-------------------------------------
Attachment: (was: metasave.patch)
> Tools to analyze and debug namenode on a production cluster
> -----------------------------------------------------------
>
> Key: HADOOP-1133
> URL: https://issues.apache.org/jira/browse/HADOOP-1133
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: metasave3.patch
>
>
> I would like to have a tool that dumps namenode data structures to a log file. This is especially useful in debugging a production cluster where attaching a debugger to the namenode is not a feasible option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1133) Tools to analyze and debug namenode
on a production cluster
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486523 ]
Hadoop QA commented on HADOOP-1133:
-----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12354889/metasave3.patch applied and successfully tested against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/525290. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
> Tools to analyze and debug namenode on a production cluster
> -----------------------------------------------------------
>
> Key: HADOOP-1133
> URL: https://issues.apache.org/jira/browse/HADOOP-1133
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: metasave3.patch
>
>
> I would like to have a tool that dumps namenode data structures to a log file. This is especially useful in debugging a production cluster where attaching a debugger to the namenode is not a feasible option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1133) Tools to analyze and debug namenode
on a production cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1133:
-------------------------------------
Attachment: metasave.patch
Here is a tool that is invoked as follows:
bin/hadoop dfsadmin -metasave <filename>
The namenode dumps its primary data structures into a file specified by the above command. It creates the file in the directory specified by a system property "hadoop.log.dir".
The tools dumps one line for each of the following:
1. Datanodes that the namenode is heartbeating with
2. Blocks that are waiting replication
3. Blocks that are currently being replicated
4. Blocks that are waiting to be deleted
Comments welcome.
> Tools to analyze and debug namenode on a production cluster
> -----------------------------------------------------------
>
> Key: HADOOP-1133
> URL: https://issues.apache.org/jira/browse/HADOOP-1133
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: metasave.patch
>
>
> I would like to have a tool that dumps namenode data structures to a log file. This is especially useful in debugging a production cluster where attaching a debugger to the namenode is not a feasible option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1133) Tools to analyze and debug namenode
on a production cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1133:
-------------------------------------
Attachment: metasave3.patch
Merged with latest trunk.
> Tools to analyze and debug namenode on a production cluster
> -----------------------------------------------------------
>
> Key: HADOOP-1133
> URL: https://issues.apache.org/jira/browse/HADOOP-1133
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: metasave3.patch
>
>
> I would like to have a tool that dumps namenode data structures to a log file. This is especially useful in debugging a production cluster where attaching a debugger to the namenode is not a feasible option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1133) Tools to analyze and debug namenode
on a production cluster
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1133:
-------------------------------------
Status: Patch Available (was: Open)
A tools that dumps namenode data structures into a log file.
> Tools to analyze and debug namenode on a production cluster
> -----------------------------------------------------------
>
> Key: HADOOP-1133
> URL: https://issues.apache.org/jira/browse/HADOOP-1133
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: metasave3.patch
>
>
> I would like to have a tool that dumps namenode data structures to a log file. This is especially useful in debugging a production cluster where attaching a debugger to the namenode is not a feasible option.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.