You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2010/06/23 02:30:52 UTC
[jira] Created: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
0.20: Block lost when multiple DNs trying to recover it to different genstamps
------------------------------------------------------------------------------
Key: HDFS-1260
URL: https://issues.apache.org/jira/browse/HDFS-1260
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.20-append
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical
Fix For: 0.20-append
Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
- FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
- we find the block file and meta file based on block ID only, without comparing gen stamp
- we rename the meta file to the new genstamp _7094
- in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
- validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882089#action_12882089 ]
dhruba borthakur commented on HDFS-1260:
----------------------------------------
This patch looks perfect to me. +1
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>
> Key: HDFS-1260
> URL: https://issues.apache.org/jira/browse/HDFS-1260
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20-append
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.20-append
>
> Attachments: hdfs-1260.txt, hdfs-1260.txt
>
>
> Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen stamp
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881490#action_12881490 ]
Todd Lipcon commented on HDFS-1260:
-----------------------------------
To confirm the suspicion above, I had the operator rename the meta block back to _7094, and the next recovery attempt succeeded.
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>
> Key: HDFS-1260
> URL: https://issues.apache.org/jira/browse/HDFS-1260
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20-append
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.20-append
>
>
> Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen stamp
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
Posted by "sam rash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881991#action_12881991 ]
sam rash commented on HDFS-1260:
--------------------------------
yea, looks good. at some point, does it make sense to move the DelayAnswer class out? it seems generally useful (not this patch, but just thinking)
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>
> Key: HDFS-1260
> URL: https://issues.apache.org/jira/browse/HDFS-1260
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20-append
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.20-append
>
> Attachments: hdfs-1260.txt, hdfs-1260.txt
>
>
> Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen stamp
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-1260:
------------------------------
Attachment: simultaneous-recoveries.txt
After months of running this test I ran into this failure attached above. One of the DNs somehow ends up with multiple meta files for the same block, but at different generation stamps.
I think the issue is in the implementation of DataNode.updateBlock(). The block passed in doesn't have a wildcard generation stamp, but we don't care - we go and find the block on disk without matching generation stamps. I think this is OK based on the validation logic - we still only move blocks forward in GS-time, and don't revert length. However, when we then call updateBlockMap() it doesn't use a wildcard generation stamp, so the block can get left in the block map with the old generation stamp. This inconsistency I think cascades into the sort of failure seen in the attached log.
I *think* the solution is:
- Change updateBlock to call updateBlockMap with a wildcard generation stamp key
- Change the interruption code to use a wildcard GS block when interrupting concurrent writers
I will make these changes and see if the rest of the unit tests still pass, then see if I can come up with a regression test.
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>
> Key: HDFS-1260
> URL: https://issues.apache.org/jira/browse/HDFS-1260
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20-append
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.20-append
>
> Attachments: hdfs-1260.txt, hdfs-1260.txt, simultaneous-recoveries.txt
>
>
> Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen stamp
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881993#action_12881993 ]
Todd Lipcon commented on HDFS-1260:
-----------------------------------
Yea, we could move it to a MockitoUtil class or something. Let's tackle that when we move all these tests forward to trunk (I plan to do that in July hopefully)
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>
> Key: HDFS-1260
> URL: https://issues.apache.org/jira/browse/HDFS-1260
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20-append
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.20-append
>
> Attachments: hdfs-1260.txt, hdfs-1260.txt
>
>
> Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen stamp
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-1260:
------------------------------
Attachment: hdfs-1260.txt
Simple patch which reorders tryUpdateBlock's verification vs rename.
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>
> Key: HDFS-1260
> URL: https://issues.apache.org/jira/browse/HDFS-1260
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20-append
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.20-append
>
> Attachments: hdfs-1260.txt
>
>
> Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen stamp
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
Posted by "sam rash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881988#action_12881988 ]
sam rash commented on HDFS-1260:
--------------------------------
oh, other than that, lgtm
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>
> Key: HDFS-1260
> URL: https://issues.apache.org/jira/browse/HDFS-1260
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20-append
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.20-append
>
> Attachments: hdfs-1260.txt
>
>
> Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen stamp
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881491#action_12881491 ]
Todd Lipcon commented on HDFS-1260:
-----------------------------------
err.,.. sorry... rename the meta block back to *_7093*
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>
> Key: HDFS-1260
> URL: https://issues.apache.org/jira/browse/HDFS-1260
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20-append
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.20-append
>
>
> Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen stamp
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881986#action_12881986 ]
Nicolas Spiegelberg commented on HDFS-1260:
-------------------------------------------
+1 on the fix. good catch
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>
> Key: HDFS-1260
> URL: https://issues.apache.org/jira/browse/HDFS-1260
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20-append
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.20-append
>
> Attachments: hdfs-1260.txt
>
>
> Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen stamp
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902652#action_12902652 ]
dhruba borthakur commented on HDFS-1260:
----------------------------------------
> However, when we then call updateBlockMap() it doesn't use a wildcard generation stamp,
yes, that seems to be a bug to me. Great catch!
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>
> Key: HDFS-1260
> URL: https://issues.apache.org/jira/browse/HDFS-1260
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20-append
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.20-append
>
> Attachments: hdfs-1260.txt, hdfs-1260.txt, simultaneous-recoveries.txt
>
>
> Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen stamp
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
Posted by "sam rash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881985#action_12881985 ]
sam rash commented on HDFS-1260:
--------------------------------
about the testing, any reason not to use one of the adapters instead of making this public?
{code}
public long nextGenerationStampForBlock(Block block) throws IOException {
{code}
sorry, i'm a stickler for visibility/encapsulation bits when i can be
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>
> Key: HDFS-1260
> URL: https://issues.apache.org/jira/browse/HDFS-1260
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20-append
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.20-append
>
> Attachments: hdfs-1260.txt
>
>
> Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen stamp
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1260) 0.20: Block lost when multiple DNs
trying to recover it to different genstamps
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-1260:
------------------------------
Attachment: hdfs-1260.txt
Here's a patch that moves the call over to the adapter. Also added a bit of javadoc to DelayAnswer
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>
> Key: HDFS-1260
> URL: https://issues.apache.org/jira/browse/HDFS-1260
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20-append
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 0.20-append
>
> Attachments: hdfs-1260.txt, hdfs-1260.txt
>
>
> Saw this issue on a cluster where some ops people were doing network changes without shutting down DNs first. So, recovery ended up getting started at multiple different DNs at the same time, and some race condition occurred that caused a block to get permanently stuck in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen stamp
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo, since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above) and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.