You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Jeff Hammerbacher (JIRA)" <ji...@apache.org> on 2010/04/29 11:55:54 UTC
[jira] Created: (HDFS-1120) Make DataNode's block-to-device
placement policy pluggable
Make DataNode's block-to-device placement policy pluggable
----------------------------------------------------------
Key: HDFS-1120
URL: https://issues.apache.org/jira/browse/HDFS-1120
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node
Reporter: Jeff Hammerbacher
As discussed on the mailing list, as the number of disk drives per server increases, it would be useful to allow the DataNode's policy for new block placement to grow in sophistication from the current round-robin strategy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1120) Make DataNode's block-to-device
placement policy pluggable
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862762#action_12862762 ]
Eli Collins commented on HDFS-1120:
-----------------------------------
This is the result of discussion on in the "dfs.data.dir" thread on general. The high-level bit is that disk utilization gets imbalanced so you get effectively fewer spindles for writes since some disks will will up much faster than others. Do you see this in your clusters?
> Make DataNode's block-to-device placement policy pluggable
> ----------------------------------------------------------
>
> Key: HDFS-1120
> URL: https://issues.apache.org/jira/browse/HDFS-1120
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Jeff Hammerbacher
>
> As discussed on the mailing list, as the number of disk drives per server increases, it would be useful to allow the DataNode's policy for new block placement to grow in sophistication from the current round-robin strategy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1120) Make DataNode's block-to-device
placement policy pluggable
Posted by "Ambikeshwar Raj Merchia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890975#action_12890975 ]
Ambikeshwar Raj Merchia commented on HDFS-1120:
-----------------------------------------------
What ever policy is developed, the following should taken into consideration:
The expected life time of the blocks.
My experience oldest data (from creation) is the first to be removed.
Usage of the blocks.
My experience newest data (from creation) is the most like to be read again.
Disk Scrub reads and validates the data. To do the balancing reading and writing are both necessary. Therefore, should the disk scrub be overloaded to do the balancing and reduce 50% of the work of balancing?
> Make DataNode's block-to-device placement policy pluggable
> ----------------------------------------------------------
>
> Key: HDFS-1120
> URL: https://issues.apache.org/jira/browse/HDFS-1120
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Jeff Hammerbacher
>
> As discussed on the mailing list, as the number of disk drives per server increases, it would be useful to allow the DataNode's policy for new block placement to grow in sophistication from the current round-robin strategy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1120) Make DataNode's block-to-device
placement policy pluggable
Posted by "Travis Crawford (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890967#action_12890967 ]
Travis Crawford commented on HDFS-1120:
---------------------------------------
Moving this comment from my duplicate.
Filing this issue in response to ``full disk woes`` on hdfs-user.
Datanodes fill their storage directories unevenly, leading to situations where certain disks are full while others are significantly less used. Users at many different sites have experienced this issue, and HDFS administrators are taking steps like:
- Manually rebalancing blocks in storage directories
- Decomissioning nodes & later readding them
There's a tradeoff between making use of all available spindles, and filling disks at the sameish rate. Possible solutions include:
- Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy environments this will still make use of all spindles, equalizing disk use over time.
- Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced in older cluster nodes.
Datanodes should actively manage their local disk so operator intervention is not needed.
> Make DataNode's block-to-device placement policy pluggable
> ----------------------------------------------------------
>
> Key: HDFS-1120
> URL: https://issues.apache.org/jira/browse/HDFS-1120
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Jeff Hammerbacher
>
> As discussed on the mailing list, as the number of disk drives per server increases, it would be useful to allow the DataNode's policy for new block placement to grow in sophistication from the current round-robin strategy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1120) Make DataNode's block-to-device
placement policy pluggable
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862675#action_12862675 ]
dhruba borthakur commented on HDFS-1120:
----------------------------------------
do you have a use-case in mind when such a policy if beneficial and why the current policy is inadequate?
is a policy that tries to keep all disks at the same usage capacity better then the round-robin one? especially when disks can die, get repaired and then put back into the same system?
> Make DataNode's block-to-device placement policy pluggable
> ----------------------------------------------------------
>
> Key: HDFS-1120
> URL: https://issues.apache.org/jira/browse/HDFS-1120
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Jeff Hammerbacher
>
> As discussed on the mailing list, as the number of disk drives per server increases, it would be useful to allow the DataNode's policy for new block placement to grow in sophistication from the current round-robin strategy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1120) Make DataNode's block-to-device
placement policy pluggable
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892257#action_12892257 ]
Steve Loughran commented on HDFS-1120:
--------------------------------------
@Ambikeshwar -you make some good points. I think we should
# start with some monitoring of what's going on, especially on 6-12 HDD servers
# do some plugin point where people can play with block placement
# Let people with time on their hands do the good plugins
For the complex decisions, you probably do want some history -filename, created, last read, owner.
I think your point about rebalancing is a good one, it may be better to schedule some rebalancing work on a DN that can spread stuff across the disks than try and do some smarter block placement decisions. If you can rebalance the disks then you can just use roundrobin or roulette placement. This handles the new-HDD problem better, and can allow for the block placer to be simple and fast, rebalancing can be done on selective nodes at a time of choosing.
> Make DataNode's block-to-device placement policy pluggable
> ----------------------------------------------------------
>
> Key: HDFS-1120
> URL: https://issues.apache.org/jira/browse/HDFS-1120
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Jeff Hammerbacher
>
> As discussed on the mailing list, as the number of disk drives per server increases, it would be useful to allow the DataNode's policy for new block placement to grow in sophistication from the current round-robin strategy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1120) Make DataNode's block-to-device
placement policy pluggable
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863740#action_12863740 ]
Steve Loughran commented on HDFS-1120:
--------------------------------------
I think the probability gets larger the more disks/server, and now that 12HDD units are coming out, you can plan to see it some time after you spec out your next datacentre.
Causes
# deletion of large block size files can leave a disk unbalanced.
# MR temp space in the same disks can fill up then free disks
# Replacement of failed HDDs leaves that disk permanently underutilised.
the third one is new; on a 12 disk server, with most of all 12 disks allocated to HDFS, one block in 12 would go to any specific disk. If one disk is replaced, it still only gets 1/12 of the blocks, even though if all the other disks were 70-80% full, its the disk with the most space. The disks would only be balanced if the new disk got more of the writes (which could have adverse consequences for future IO rates), or some rebalancing on a single machine moves data from one disk to another (or to be precise, copies, validates the block checksums, then deletes).
I actually think HDFS-1121 should come first: provide a way of measuring the distribution on disks on a single server. Once we have the data we can start worrying about ways to correct any distribution issues.
> Make DataNode's block-to-device placement policy pluggable
> ----------------------------------------------------------
>
> Key: HDFS-1120
> URL: https://issues.apache.org/jira/browse/HDFS-1120
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Jeff Hammerbacher
>
> As discussed on the mailing list, as the number of disk drives per server increases, it would be useful to allow the DataNode's policy for new block placement to grow in sophistication from the current round-robin strategy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1120) Make DataNode's block-to-device
placement policy pluggable
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863104#action_12863104 ]
dhruba borthakur commented on HDFS-1120:
----------------------------------------
Agreed, it is a possible scenario. I have not looked deeply enough to figure out if this scenario occurs in our cluster.
> Make DataNode's block-to-device placement policy pluggable
> ----------------------------------------------------------
>
> Key: HDFS-1120
> URL: https://issues.apache.org/jira/browse/HDFS-1120
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Reporter: Jeff Hammerbacher
>
> As discussed on the mailing list, as the number of disk drives per server increases, it would be useful to allow the DataNode's policy for new block placement to grow in sophistication from the current round-robin strategy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.