You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Travis Crawford (JIRA)" <ji...@apache.org> on 2010/07/22 01:47:50 UTC
[jira] Created: (HDFS-1312) Datanode storage directories fill
unevenly
Datanode storage directories fill unevenly
------------------------------------------
Key: HDFS-1312
URL: https://issues.apache.org/jira/browse/HDFS-1312
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Reporter: Travis Crawford
Filing this issue in response to ``full disk woes`` on hdfs-user.
Datanodes fill their storage directories unevenly, leading to situations where certain disks are full while others are significantly less used. Users at many different sites have experienced this issue, and HDFS administrators are taking steps like:
- Manually rebalancing blocks in storage directories
- Decomissioning nodes & later readding them
There's a tradeoff between making use of all available spindles, and filling disks at the sameish rate. Possible solutions include:
- Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy environments this will still make use of all spindles, equalizing disk use over time.
- Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced in older cluster nodes.
Datanodes should actively manage their local disk so operator intervention is not needed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-1312) Datanode storage directories fill
unevenly
Posted by "Travis Crawford (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Travis Crawford resolved HDFS-1312.
-----------------------------------
Resolution: Duplicate
HDFS-1120 looks like a better place to track this issue. Marking this as a duplicate.
> Datanode storage directories fill unevenly
> ------------------------------------------
>
> Key: HDFS-1312
> URL: https://issues.apache.org/jira/browse/HDFS-1312
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Travis Crawford
>
> Filing this issue in response to ``full disk woes`` on hdfs-user.
> Datanodes fill their storage directories unevenly, leading to situations where certain disks are full while others are significantly less used. Users at many different sites have experienced this issue, and HDFS administrators are taking steps like:
> - Manually rebalancing blocks in storage directories
> - Decomissioning nodes & later readding them
> There's a tradeoff between making use of all available spindles, and filling disks at the sameish rate. Possible solutions include:
> - Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy environments this will still make use of all spindles, equalizing disk use over time.
> - Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced in older cluster nodes.
> Datanodes should actively manage their local disk so operator intervention is not needed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1312) Datanode storage directories fill
unevenly
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892258#action_12892258 ]
Steve Loughran commented on HDFS-1312:
--------------------------------------
HDFS-1120 is part of the solution, but some rebalancing util is needed to. Maybe this issue should be re-opened as the underlying problem, HDFS-1120 and HDFS-1121 as parts of the fix, along with "add a way to rebalance HDDs on a single DN"
> Datanode storage directories fill unevenly
> ------------------------------------------
>
> Key: HDFS-1312
> URL: https://issues.apache.org/jira/browse/HDFS-1312
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Travis Crawford
>
> Filing this issue in response to ``full disk woes`` on hdfs-user.
> Datanodes fill their storage directories unevenly, leading to situations where certain disks are full while others are significantly less used. Users at many different sites have experienced this issue, and HDFS administrators are taking steps like:
> - Manually rebalancing blocks in storage directories
> - Decomissioning nodes & later readding them
> There's a tradeoff between making use of all available spindles, and filling disks at the sameish rate. Possible solutions include:
> - Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy environments this will still make use of all spindles, equalizing disk use over time.
> - Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced in older cluster nodes.
> Datanodes should actively manage their local disk so operator intervention is not needed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1312) Datanode storage directories fill
unevenly
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890956#action_12890956 ]
dhruba borthakur commented on HDFS-1312:
----------------------------------------
rebalance blocks locally by each individual datanode sounds like a good idea. we have seen this problem in our cluster too.
> Datanode storage directories fill unevenly
> ------------------------------------------
>
> Key: HDFS-1312
> URL: https://issues.apache.org/jira/browse/HDFS-1312
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Travis Crawford
>
> Filing this issue in response to ``full disk woes`` on hdfs-user.
> Datanodes fill their storage directories unevenly, leading to situations where certain disks are full while others are significantly less used. Users at many different sites have experienced this issue, and HDFS administrators are taking steps like:
> - Manually rebalancing blocks in storage directories
> - Decomissioning nodes & later readding them
> There's a tradeoff between making use of all available spindles, and filling disks at the sameish rate. Possible solutions include:
> - Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy environments this will still make use of all spindles, equalizing disk use over time.
> - Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced in older cluster nodes.
> Datanodes should actively manage their local disk so operator intervention is not needed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1312) Datanode storage directories fill
unevenly
Posted by "Scott Carey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890961#action_12890961 ]
Scott Carey commented on HDFS-1312:
-----------------------------------
a roulette-like algorithm for choosing which disk to write a block to should be relatively easy and keep the disks generally balanced except when major changes such as adding a new disk occur.
I think the safest default option is to make the relative odds of choosing a disk equal to the free space available. The disk placement code already has access to that information and checks it to see if the disk is full. Changing it from round-robin to weighted is straightforward unless this needs to be plugable.
A larger disk will get more writes, but there is no avoiding that if you want them to be balanced. Balancing after placement causes more overall I/O then placing it right the first time.
Rebalancing will always be needed from time for other reasons however.
> Datanode storage directories fill unevenly
> ------------------------------------------
>
> Key: HDFS-1312
> URL: https://issues.apache.org/jira/browse/HDFS-1312
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Reporter: Travis Crawford
>
> Filing this issue in response to ``full disk woes`` on hdfs-user.
> Datanodes fill their storage directories unevenly, leading to situations where certain disks are full while others are significantly less used. Users at many different sites have experienced this issue, and HDFS administrators are taking steps like:
> - Manually rebalancing blocks in storage directories
> - Decomissioning nodes & later readding them
> There's a tradeoff between making use of all available spindles, and filling disks at the sameish rate. Possible solutions include:
> - Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy environments this will still make use of all spindles, equalizing disk use over time.
> - Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced in older cluster nodes.
> Datanodes should actively manage their local disk so operator intervention is not needed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.