You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Robert Chansler (JIRA)" <ji...@apache.org> on 2008/10/31 01:44:46 UTC
[jira] Created: (HADOOP-4556) Block went missing
Block went missing
------------------
Key: HADOOP-4556
URL: https://issues.apache.org/jira/browse/HADOOP-4556
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.17.2
Reporter: Robert Chansler
Assignee: Hairong Kuang
Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644431#action_12644431 ]
Raghu Angadi commented on HADOOP-4556:
--------------------------------------
The "main cause" (among others) seems to be because of incomplete clean up when a datanode is marked dead.
Specifically, the pending requests for block deletions don't seem to be removed when a node is marked dead.. These deletion requests might be sent after the datanode registers back again... but the state might be very different that time.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644434#action_12644434 ]
Raghu Angadi commented on HADOOP-4556:
--------------------------------------
yes. E.g. :
- P and Q have a block B. Say B is over replicated to R and P is asked to delete the block.
- Now Q and R have the block and there is a pending delete-request to P.
- Now P is marked dead and P comes back up again.
- block will become over replicated and NN asks Q to delete it.
- Now the block will be delete from both P and Q! Block is under replicated.
Does it seem right? IOW, why do we not want to remove the pending deletion requests?
The above example shows loss of one replica unintentionally. The real example is lot crazier in terms of how the DNs were maked dead and came back alive and we ended up losing all the replicas. I don't know yet why the heartbeat was lost.
Note that this in no way means we should not fix HADOOP-4540. These are two different problems (or two parts of a problem).
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646360#action_12646360 ]
Hairong Kuang commented on HADOOP-4556:
---------------------------------------
> -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
This is not right. I ran "ant javadoc" with and without the patch. I did not see that the patch added any additional javadoc warning.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4556) Block went missing
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4556:
----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
The patch is manually tested. The javadoc warning is irrelevant of this patch.
I just committed this.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch, staleInvalidates2-br18.patch, staleInvalidates2.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646346#action_12646346 ]
Hairong Kuang commented on HADOOP-4556:
---------------------------------------
Having addToInvalidatesNoLog is to avoid an extra log and allow the calling function to log the reason why a block gets invalid.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645653#action_12645653 ]
Raghu Angadi commented on HADOOP-4556:
--------------------------------------
Couple of minor comments :
- The debug comment removed while choosing excess replicas can be retained and changed to INFO level. This is a pretty useful log.
- in removeFromInvalidates(), we need to log only if remove() returns true. Usually it will be false.
I am trying to see one of the actual cases seen will be fixed by this patch.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: staleInvalidates.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4556) Block went missing
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4556:
----------------------------------
Attachment: staleInvalidates2-br18.patch
Patch for branch 0.18.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch, staleInvalidates2-br18.patch, staleInvalidates2.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4556) Block went missing
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-4556:
---------------------------------
Attachment: missing-block-example.txt
Trace of a block that got all of its replicas deleted (courtesy Rob Chansler).
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4556) Block went missing
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4556:
----------------------------------
Status: Patch Available (was: Open)
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4556) Block went missing
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4556:
----------------------------------
Attachment: staleInvalidates2.patch
The attached patch put the cleaning of two invalidate sets in one place when a datanode is removed.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch, staleInvalidates2.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4556) Block went missing
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4556:
----------------------------------
Attachment: staleInvalidates1.patch
The patch incorporates Raghu's comment.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646345#action_12646345 ]
Raghu Angadi commented on HADOOP-4556:
--------------------------------------
> 1. removeStoredBlock makes sure that the replica also gets removed from recentInvalidatesSet;
the patch removes it from the other set.
So we have two sets : one in dn and one in FSNamesystem. These two are cleared in two different places (and two different reasons): dn's is cleared when a datanode is marked dead and FSNamesystem's is cleared when a block is deleted.
I think for consistency and clarity, both should be cleared in one place in the code.
The patch as is I think fixes the problem.. so if we don't want to change the patch, it might be ok for 0.18. But for 19 and trunk at least I think it is better to implement this policy in one place.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646341#action_12646341 ]
Hairong Kuang commented on HADOOP-4556:
---------------------------------------
recentInvalidatesSet is cleared in removeStoredBlock.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644439#action_12644439 ]
dhruba borthakur commented on HADOOP-4556:
------------------------------------------
Ok, got it. This is a race between the block report in Step 3 ("Now P is marked dead and P comes back up again") and the block-delete-call that the namenode issued in Step 1. The block report inserted a block in the blocksmap that is going to be deleted from the datanode.
Two options come to mind:
1. On every block report processing, the NN clears out blocks from invalidateSets that belong to that datanode. This might consume non-trivial amount of CPU on the namenode.
2. The datanode sends its current time on every block report, let call this as the reportStamp. The NN stores the reportStamp in the DatanodeDescriptor and sends it back to the DN on every block deletion message. The Datanode process a delete-block request only if the reportStamp in the delete-block-message is greater than the reportStamp it used in the last block report that it send out.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646372#action_12646372 ]
Raghu Angadi commented on HADOOP-4556:
--------------------------------------
I tested the patch with an artificial test that brings datanodes down and brings them up at different times among other things.
on trunk, the block loses all the replicas and with the patch it does not.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646332#action_12646332 ]
Raghu Angadi commented on HADOOP-4556:
--------------------------------------
> 2. When removing a datanode, reset its scheduled invalidate blocks.
There is also 'recentIvalidateSets'. When NN deletes a block, it first goes into 'recentInvalidateSets', then the replication monitor thread moves this request to datanode's invalidateSet. I don't know why we have this two step process, but looks like this patch needs to clear the datanode's set in recentInvalidateSets as well.
I don't think 'addToInvalidatesNoLog' is necessary. The extra log is fine and will make it consistent as well.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4556) Block went missing
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4556:
----------------------------------
Attachment: staleInvalidates.patch
This patch fixes the stale invalidates problem by
1. removeStoredBlock makes sure that the replica also gets removed from recentInvalidatesSet;
2. When removing a datanode, reset its scheduled invalidate blocks;
3. Fix a typo: proccessOverReplicatedBlock +> processOverReplicatedBlock
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: staleInvalidates.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646416#action_12646416 ]
Hairong Kuang commented on HADOOP-4556:
---------------------------------------
ant test-core result:
BUILD SUCCESSFUL
Total time: 116 minutes 32 seconds
ant test-patch result:
[exec] -1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch, staleInvalidates2-br18.patch, staleInvalidates2.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645953#action_12645953 ]
Hadoop QA commented on HADOOP-4556:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12393522/staleInvalidates1.patch
against trunk revision 712339.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3554/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3554/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3554/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3554/console
This message is automatically generated.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4556) Block went missing
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4556:
----------------------------------
Fix Version/s: 0.18.3
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: staleInvalidates.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644433#action_12644433 ]
dhruba borthakur commented on HADOOP-4556:
------------------------------------------
> These deletion requests might be sent after the datanode registers back again
But then this will delete a block that was anyway supposed to have been deleted much earlier. can this really cause the bug described?
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4556) Block went missing
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646382#action_12646382 ]
Raghu Angadi commented on HADOOP-4556:
--------------------------------------
Thanks Hairong. +1. patch looks good to me.
> Block went missing
> ------------------
>
> Key: HADOOP-4556
> URL: https://issues.apache.org/jira/browse/HADOOP-4556
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.2
> Reporter: Robert Chansler
> Assignee: Hairong Kuang
> Fix For: 0.18.3
>
> Attachments: missing-block-example.txt, staleInvalidates.patch, staleInvalidates1.patch, staleInvalidates2.patch
>
>
> Suspicion that all replicas of a block were marked for deletion. (Don't panic, investigation underway.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.