You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2006/05/01 20:10:46 UTC
[jira] Created: (HADOOP-178) piggyback block work requests to
heartbeats and move block replication/deletion startup delay from datanodes
to namenode
piggyback block work requests to heartbeats and move block replication/deletion startup delay from datanodes to namenode
------------------------------------------------------------------------------------------------------------------------
Key: HADOOP-178
URL: http://issues.apache.org/jira/browse/HADOOP-178
Project: Hadoop
Type: Improvement
Components: dfs
Reporter: Hairong Kuang
Assigned to: Hairong Kuang
Currently each datanode sends at least two messages to namenode within a heartbeat interval. One is a heartbeat message and another is block work request. By piggybacking the block work request to a heartbeat can greatly cut the number of messages between a datanode and the namenode.
Secondly each datanode waits for a configurable "StartupPeriod" before it sends a block work request in order to avoid uneccessary block replication at startup time. But if the namenode starts much later than datanodes, this scheme does not work. Furthermore, the namenode has more information to decide when to send block work to datanodes. For example, all datanodes send block reports etc. It is more resonable to move the startup delay from datanodes to the namenode
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-178) piggyback block work requests to
heartbeats and move block replication/deletion startup delay from datanodes
to namenode
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-178?page=all ]
Doug Cutting resolved HADOOP-178:
---------------------------------
Fix Version: 0.2
Resolution: Fixed
I just committed this. Thanks, Hairong!
> piggyback block work requests to heartbeats and move block replication/deletion startup delay from datanodes to namenode
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-178
> URL: http://issues.apache.org/jira/browse/HADOOP-178
> Project: Hadoop
> Type: Improvement
> Components: dfs
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.2
> Attachments: startupDelay.patch
>
> Currently each datanode sends at least two messages to namenode within a heartbeat interval. One is a heartbeat message and another is block work request. By piggybacking the block work request to a heartbeat can greatly cut the number of messages between a datanode and the namenode.
> Secondly each datanode waits for a configurable "StartupPeriod" before it sends a block work request in order to avoid uneccessary block replication at startup time. But if the namenode starts much later than datanodes, this scheme does not work. Furthermore, the namenode has more information to decide when to send block work to datanodes. For example, all datanodes send block reports etc. It is more resonable to move the startup delay from datanodes to the namenode
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-178) piggyback block work requests to
heartbeats and move block replication/deletion startup delay from datanodes
to namenode
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-178?page=all ]
Hairong Kuang updated HADOOP-178:
---------------------------------
Attachment: startupDelay.patch
I made the changes described in the issue report. In addition, I made the granularity of locking on receivedBlockList to be smaller, i.e. the code synchronizes on receivedBlockList only when reading/writing to the list. Also there seemed to be a bug on line 174 in the patch when calculating waittime. So I changed "now" to be System.currentTimeMillis().
> piggyback block work requests to heartbeats and move block replication/deletion startup delay from datanodes to namenode
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-178
> URL: http://issues.apache.org/jira/browse/HADOOP-178
> Project: Hadoop
> Type: Improvement
> Components: dfs
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Attachments: startupDelay.patch
>
> Currently each datanode sends at least two messages to namenode within a heartbeat interval. One is a heartbeat message and another is block work request. By piggybacking the block work request to a heartbeat can greatly cut the number of messages between a datanode and the namenode.
> Secondly each datanode waits for a configurable "StartupPeriod" before it sends a block work request in order to avoid uneccessary block replication at startup time. But if the namenode starts much later than datanodes, this scheme does not work. Furthermore, the namenode has more information to decide when to send block work to datanodes. For example, all datanodes send block reports etc. It is more resonable to move the startup delay from datanodes to the namenode
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira