You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2007/09/06 19:04:35 UTC
[jira] Created: (HADOOP-1846) DatanodeReport should distinguish
live datanodes from dead datanodes
DatanodeReport should distinguish live datanodes from dead datanodes
--------------------------------------------------------------------
Key: HADOOP-1846
URL: https://issues.apache.org/jira/browse/HADOOP-1846
Project: Hadoop
Issue Type: Bug
Components: dfs
Affects Versions: 0.14.1
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Fix For: 0.15.0
DatanodeReport returns both live and dead datanodes and there is no way to distinguish live nodes from dead nodes. But some applications are interested in only live datanodes. I propose that instead of haveing one datanodeReport, dfs supports two reports: liveDatanodeReport and deadDatanodeReport.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1846) DatanodeReport should distinguish
live datanodes from dead datanodes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12525852 ]
Hairong Kuang commented on HADOOP-1846:
---------------------------------------
I plan to fix the problem by changing the datanodeReport protocol to be
public DatanodeInfo[] getDatanodeReport(NodeType type) throws IOException;
The value of type could be ALL: return all data nodes
LIVE: return live data nodes
DEAD: return dea data nodes
> DatanodeReport should distinguish live datanodes from dead datanodes
> --------------------------------------------------------------------
>
> Key: HADOOP-1846
> URL: https://issues.apache.org/jira/browse/HADOOP-1846
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
>
> DatanodeReport returns both live and dead datanodes and there is no way to distinguish live nodes from dead nodes. But some applications are interested in only live datanodes. I propose that instead of haveing one datanodeReport, dfs supports two reports: liveDatanodeReport and deadDatanodeReport.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1846) DatanodeReport should distinguish
live datanodes from dead datanodes
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526561 ]
Sameer Paranjpye commented on HADOOP-1846:
------------------------------------------
If it is not in hadoop-default.xml then why have it at all? Better to have a static final instead.
> DatanodeReport should distinguish live datanodes from dead datanodes
> --------------------------------------------------------------------
>
> Key: HADOOP-1846
> URL: https://issues.apache.org/jira/browse/HADOOP-1846
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanodeReport.patch
>
>
> DatanodeReport returns both live and dead datanodes and there is no way to distinguish live nodes from dead nodes. But some applications are interested in only live datanodes. I propose that instead of haveing one datanodeReport, dfs supports two reports: liveDatanodeReport and deadDatanodeReport.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1846) DatanodeReport should distinguish
live datanodes from dead datanodes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1846:
----------------------------------
Attachment: datanodeReport.patch
> DatanodeReport should distinguish live datanodes from dead datanodes
> --------------------------------------------------------------------
>
> Key: HADOOP-1846
> URL: https://issues.apache.org/jira/browse/HADOOP-1846
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanodeReport.patch
>
>
> DatanodeReport returns both live and dead datanodes and there is no way to distinguish live nodes from dead nodes. But some applications are interested in only live datanodes. I propose that instead of haveing one datanodeReport, dfs supports two reports: liveDatanodeReport and deadDatanodeReport.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1846) DatanodeReport should distinguish
live datanodes from dead datanodes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526539 ]
dhruba borthakur commented on HADOOP-1846:
------------------------------------------
Code look good. Two minor comments:
1. In MiniDFSCluster. waitActive(), we should probably be invoking client.datanodeReport(ALL) rather than client.datanodeReport(LIVE).
2. This patch introduces a new conf variable called heartbeat.recheck.interval.
is this really needed?
> DatanodeReport should distinguish live datanodes from dead datanodes
> --------------------------------------------------------------------
>
> Key: HADOOP-1846
> URL: https://issues.apache.org/jira/browse/HADOOP-1846
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanodeReport.patch
>
>
> DatanodeReport returns both live and dead datanodes and there is no way to distinguish live nodes from dead nodes. But some applications are interested in only live datanodes. I propose that instead of haveing one datanodeReport, dfs supports two reports: liveDatanodeReport and deadDatanodeReport.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1846) DatanodeReport should distinguish
live datanodes from dead datanodes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526560 ]
Hairong Kuang commented on HADOOP-1846:
---------------------------------------
> This patch introduces a new conf variable called heartbeat.recheck.interval.
This conf variable is to make TestDatanodeReport to run quicker. A datanode won't be declared dead after a name node has not received its heartbeat for 2*heartbeat.recheck.interval+10*heartbeat interval. The heartbeat recheck interval was set to be 5 minutes. In the test, I set it to be 0.5s. This config variable is not intended for admin usage so it is not in hadoop-default.xml.
> DatanodeReport should distinguish live datanodes from dead datanodes
> --------------------------------------------------------------------
>
> Key: HADOOP-1846
> URL: https://issues.apache.org/jira/browse/HADOOP-1846
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanodeReport.patch
>
>
> DatanodeReport returns both live and dead datanodes and there is no way to distinguish live nodes from dead nodes. But some applications are interested in only live datanodes. I propose that instead of haveing one datanodeReport, dfs supports two reports: liveDatanodeReport and deadDatanodeReport.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1846) DatanodeReport should distinguish
live datanodes from dead datanodes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1846:
-------------------------------------
Status: Patch Available (was: Open)
> DatanodeReport should distinguish live datanodes from dead datanodes
> --------------------------------------------------------------------
>
> Key: HADOOP-1846
> URL: https://issues.apache.org/jira/browse/HADOOP-1846
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanodeReport.patch, datanodeReport.patch
>
>
> DatanodeReport returns both live and dead datanodes and there is no way to distinguish live nodes from dead nodes. But some applications are interested in only live datanodes. I propose that instead of haveing one datanodeReport, dfs supports two reports: liveDatanodeReport and deadDatanodeReport.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1846) DatanodeReport should distinguish
live datanodes from dead datanodes
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-1846:
----------------------------------
Attachment: datanodeReport.patch
This patch includes a change to MiniDFSCluster.waitActive(). It uses datanodeReport to get all live datanodes. It waits until all datanodes are alive. It also adds one more testcase to TestDatanodeReport.
> DatanodeReport should distinguish live datanodes from dead datanodes
> --------------------------------------------------------------------
>
> Key: HADOOP-1846
> URL: https://issues.apache.org/jira/browse/HADOOP-1846
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanodeReport.patch, datanodeReport.patch
>
>
> DatanodeReport returns both live and dead datanodes and there is no way to distinguish live nodes from dead nodes. But some applications are interested in only live datanodes. I propose that instead of haveing one datanodeReport, dfs supports two reports: liveDatanodeReport and deadDatanodeReport.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1846) DatanodeReport should distinguish
live datanodes from dead datanodes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1846:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I bumped up the ClientProtocol version.
Thanks Hairong.
> DatanodeReport should distinguish live datanodes from dead datanodes
> --------------------------------------------------------------------
>
> Key: HADOOP-1846
> URL: https://issues.apache.org/jira/browse/HADOOP-1846
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanodeReport.patch, datanodeReport.patch
>
>
> DatanodeReport returns both live and dead datanodes and there is no way to distinguish live nodes from dead nodes. But some applications are interested in only live datanodes. I propose that instead of haveing one datanodeReport, dfs supports two reports: liveDatanodeReport and deadDatanodeReport.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1846) DatanodeReport should distinguish
live datanodes from dead datanodes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526984 ]
Hudson commented on HADOOP-1846:
--------------------------------
Integrated in Hadoop-Nightly #234 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/234/])
> DatanodeReport should distinguish live datanodes from dead datanodes
> --------------------------------------------------------------------
>
> Key: HADOOP-1846
> URL: https://issues.apache.org/jira/browse/HADOOP-1846
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanodeReport.patch, datanodeReport.patch
>
>
> DatanodeReport returns both live and dead datanodes and there is no way to distinguish live nodes from dead nodes. But some applications are interested in only live datanodes. I propose that instead of haveing one datanodeReport, dfs supports two reports: liveDatanodeReport and deadDatanodeReport.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1846) DatanodeReport should distinguish
live datanodes from dead datanodes
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526906 ]
dhruba borthakur commented on HADOOP-1846:
------------------------------------------
+1. Looks good.
> DatanodeReport should distinguish live datanodes from dead datanodes
> --------------------------------------------------------------------
>
> Key: HADOOP-1846
> URL: https://issues.apache.org/jira/browse/HADOOP-1846
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanodeReport.patch, datanodeReport.patch
>
>
> DatanodeReport returns both live and dead datanodes and there is no way to distinguish live nodes from dead nodes. But some applications are interested in only live datanodes. I propose that instead of haveing one datanodeReport, dfs supports two reports: liveDatanodeReport and deadDatanodeReport.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1846) DatanodeReport should distinguish
live datanodes from dead datanodes
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526565 ]
Doug Cutting commented on HADOOP-1846:
--------------------------------------
> If it is not in hadoop-default.xml then why have it at all? Better to have a static final instead.
It can't be a static final if we still want to change it to make unit tests run faster, right?
I think it's okay to have parameters that are undocumented in hadoop-default.xml. These are akin to package-private fields: a useful way to share implementation internals without encouraging users to muck with them. Once something goes into hadoop-default.xml then it's public and we should continue to support it in future releases.
An alternative might be to use a package-private field or method, provided the unit test is in the same package as the parameter in question.
> DatanodeReport should distinguish live datanodes from dead datanodes
> --------------------------------------------------------------------
>
> Key: HADOOP-1846
> URL: https://issues.apache.org/jira/browse/HADOOP-1846
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.1
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
> Attachments: datanodeReport.patch
>
>
> DatanodeReport returns both live and dead datanodes and there is no way to distinguish live nodes from dead nodes. But some applications are interested in only live datanodes. I propose that instead of haveing one datanodeReport, dfs supports two reports: liveDatanodeReport and deadDatanodeReport.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.