You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Mike Cafarella (JIRA)" <ji...@apache.org> on 2006/02/07 19:05:57 UTC
[jira] Created: (HADOOP-26) DFS node choice doesn't take available
space into account effectively
DFS node choice doesn't take available space into account effectively
---------------------------------------------------------------------
Key: HADOOP-26
URL: http://issues.apache.org/jira/browse/HADOOP-26
Project: Hadoop
Type: Bug
Components: dfs
Reporter: Mike Cafarella
Assigned to: Mike Cafarella
We used to have node-allocation be sensitive to available disk space. It turned out that
this was a little accident-prone and hard to debug on then-available machines, so I removed it.
DFS is mature enough now that this needs to come back and work properly. A node with less
than X bytes should never accept a new block for allocation.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-26) DFS node choice doesn't take available
space into account effectively
Posted by "Mike Cafarella (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-26?page=all ]
Mike Cafarella updated HADOOP-26:
---------------------------------
Attachment: patch.targetchoice
Patch for this problem.
> DFS node choice doesn't take available space into account effectively
> ---------------------------------------------------------------------
>
> Key: HADOOP-26
> URL: http://issues.apache.org/jira/browse/HADOOP-26
> Project: Hadoop
> Type: Bug
> Components: dfs
> Reporter: Mike Cafarella
> Assignee: Mike Cafarella
> Attachments: patch.targetchoice
>
> We used to have node-allocation be sensitive to available disk space. It turned out that
> this was a little accident-prone and hard to debug on then-available machines, so I removed it.
> DFS is mature enough now that this needs to come back and work properly. A node with less
> than X bytes should never accept a new block for allocation.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-26) DFS node choice doesn't take
available space into account effectively
Posted by "Mike Cafarella (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-26?page=comments#action_12365457 ]
Mike Cafarella commented on HADOOP-26:
--------------------------------------
OK, the above comment is a little misleading.
We used to have a difference space-management system in place, which would set
certain nodes off-limits if they didn't have enough space. Now we choose nodes probabilistically,
with more weight given to nodes with more space.
That's probably a little too cute than is necessary. We should just have hard limits on how
much a given node will take before it complains and never receives further node allocations.
> DFS node choice doesn't take available space into account effectively
> ---------------------------------------------------------------------
>
> Key: HADOOP-26
> URL: http://issues.apache.org/jira/browse/HADOOP-26
> Project: Hadoop
> Type: Bug
> Components: dfs
> Reporter: Mike Cafarella
> Assignee: Mike Cafarella
>
> We used to have node-allocation be sensitive to available disk space. It turned out that
> this was a little accident-prone and hard to debug on then-available machines, so I removed it.
> DFS is mature enough now that this needs to come back and work properly. A node with less
> than X bytes should never accept a new block for allocation.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Closed: (HADOOP-26) DFS node choice doesn't take available
space into account effectively
Posted by "Mike Cafarella (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-26?page=all ]
Mike Cafarella closed HADOOP-26:
--------------------------------
Resolution: Fixed
Tested and fixed, in revision 383247.
> DFS node choice doesn't take available space into account effectively
> ---------------------------------------------------------------------
>
> Key: HADOOP-26
> URL: http://issues.apache.org/jira/browse/HADOOP-26
> Project: Hadoop
> Type: Bug
> Components: dfs
> Reporter: Mike Cafarella
> Assignee: Mike Cafarella
> Attachments: patch.targetchoice
>
> We used to have node-allocation be sensitive to available disk space. It turned out that
> this was a little accident-prone and hard to debug on then-available machines, so I removed it.
> DFS is mature enough now that this needs to come back and work properly. A node with less
> than X bytes should never accept a new block for allocation.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira