You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Harsh J Chouraria (JIRA)" <ji...@apache.org> on 2011/04/04 20:08:06 UTC
[jira] [Created] (HDFS-1804) Modify or add a new block-volume
device choosing policy that looks at free space
Modify or add a new block-volume device choosing policy that looks at free space
--------------------------------------------------------------------------------
Key: HDFS-1804
URL: https://issues.apache.org/jira/browse/HDFS-1804
Project: Hadoop HDFS
Issue Type: New Feature
Components: data-node
Reporter: Harsh J Chouraria
Priority: Minor
Fix For: 0.23.0
HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the vanilla RoundRobin as its default.
An additional implementation that also takes into consideration the free space remaining on the disk (or other params) should be a good addition as an alternative to vanilla RR.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1804) Modify or add a new block-volume
device choosing policy that looks at free space
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019221#comment-13019221 ]
dhruba borthakur commented on HDFS-1804:
----------------------------------------
Hi doug, I think just a biased round-robin (without respect to available free space) could be problematic, isn't it?
> Modify or add a new block-volume device choosing policy that looks at free space
> --------------------------------------------------------------------------------
>
> Key: HDFS-1804
> URL: https://issues.apache.org/jira/browse/HDFS-1804
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node
> Reporter: Harsh J Chouraria
> Priority: Minor
> Fix For: 0.23.0
>
>
> HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the vanilla RoundRobin as its default.
> An additional implementation that also takes into consideration the free space remaining on the disk (or other params) should be a good addition as an alternative to vanilla RR.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1804) Modify or add a new block-volume
device choosing policy that looks at free space
Posted by "Harsh J Chouraria (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015883#comment-13015883 ]
Harsh J Chouraria commented on HDFS-1804:
-----------------------------------------
Would using a k-smallest algorithm on the volumes list, along with a round robin after that be a good way to do this?
> Modify or add a new block-volume device choosing policy that looks at free space
> --------------------------------------------------------------------------------
>
> Key: HDFS-1804
> URL: https://issues.apache.org/jira/browse/HDFS-1804
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node
> Reporter: Harsh J Chouraria
> Priority: Minor
> Fix For: 0.23.0
>
>
> HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the vanilla RoundRobin as its default.
> An additional implementation that also takes into consideration the free space remaining on the disk (or other params) should be a good addition as an alternative to vanilla RR.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1804) Modify or add a new block-volume
device choosing policy that looks at free space
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020367#comment-13020367 ]
Doug Cutting commented on HDFS-1804:
------------------------------------
Dhruba, I meant biased by available free space. Randomly biased by free space would, as Todd suggests, not spread load as evenly between devices as would round-robin. So, if disk A has twice the free space of disks B and C then allocations should ideally go {A,B,A,C,A,B,A,C...} rather than a stateless probabilistic allocation which would often send repeated allocations to A, impacting throughput.
> Modify or add a new block-volume device choosing policy that looks at free space
> --------------------------------------------------------------------------------
>
> Key: HDFS-1804
> URL: https://issues.apache.org/jira/browse/HDFS-1804
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node
> Reporter: Harsh J Chouraria
> Priority: Minor
> Fix For: 0.23.0
>
>
> HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the vanilla RoundRobin as its default.
> An additional implementation that also takes into consideration the free space remaining on the disk (or other params) should be a good addition as an alternative to vanilla RR.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1804) Modify or add a new block-volume
device choosing policy that looks at free space
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated HDFS-1804:
--------------------------
Fix Version/s: (was: 0.24.0)
> Modify or add a new block-volume device choosing policy that looks at free space
> --------------------------------------------------------------------------------
>
> Key: HDFS-1804
> URL: https://issues.apache.org/jira/browse/HDFS-1804
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node
> Reporter: Harsh J
> Priority: Minor
> Labels: newbie
>
> HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the vanilla RoundRobin as its default.
> An additional implementation that also takes into consideration the free space remaining on the disk (or other params) should be a good addition as an alternative to vanilla RR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1804) Modify or add a new block-volume
device choosing policy that looks at free space
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015956#comment-13015956 ]
Todd Lipcon commented on HDFS-1804:
-----------------------------------
One way might be to simply choose each disk with probability relative to the amount of available space on it. So, if one disk has 2T free and another has 1T free, choose the 2T drive 2/3 of the time. Only downside is that, if two disks have an equal amount of free space, it would be better to degenerate to round-robin rather than random selection.
> Modify or add a new block-volume device choosing policy that looks at free space
> --------------------------------------------------------------------------------
>
> Key: HDFS-1804
> URL: https://issues.apache.org/jira/browse/HDFS-1804
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node
> Reporter: Harsh J Chouraria
> Priority: Minor
> Fix For: 0.23.0
>
>
> HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the vanilla RoundRobin as its default.
> An additional implementation that also takes into consideration the free space remaining on the disk (or other params) should be a good addition as an alternative to vanilla RR.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1804) Modify or add a new block-volume
device choosing policy that looks at free space
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-1804:
---------------------------------
Labels: newbie (was: )
> Modify or add a new block-volume device choosing policy that looks at free space
> --------------------------------------------------------------------------------
>
> Key: HDFS-1804
> URL: https://issues.apache.org/jira/browse/HDFS-1804
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node
> Reporter: Harsh J
> Priority: Minor
> Labels: newbie
> Fix For: 0.23.0
>
>
> HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the vanilla RoundRobin as its default.
> An additional implementation that also takes into consideration the free space remaining on the disk (or other params) should be a good addition as an alternative to vanilla RR.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1804) Modify or add a new block-volume
device choosing policy that looks at free space
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016005#comment-13016005 ]
Doug Cutting commented on HDFS-1804:
------------------------------------
Maybe a biased round robin? E.g., you might track the percentage of times that each had been allocated from and allocate from the node whose percentage is farthest from it's desired probability.
> Modify or add a new block-volume device choosing policy that looks at free space
> --------------------------------------------------------------------------------
>
> Key: HDFS-1804
> URL: https://issues.apache.org/jira/browse/HDFS-1804
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node
> Reporter: Harsh J Chouraria
> Priority: Minor
> Fix For: 0.23.0
>
>
> HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the vanilla RoundRobin as its default.
> An additional implementation that also takes into consideration the free space remaining on the disk (or other params) should be a good addition as an alternative to vanilla RR.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira