You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2007/03/20 17:52:32 UTC
[jira] Created: (HADOOP-1135) A block report processing may
incorrect cause the namenode to delete blocks
A block report processing may incorrect cause the namenode to delete blocks
----------------------------------------------------------------------------
Key: HADOOP-1135
URL: https://issues.apache.org/jira/browse/HADOOP-1135
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter: dhruba borthakur
Assigned To: dhruba borthakur
When a block report arrives at the namenode, the namenode goes through all the blocks on that datanode. If a block is not valid it is marked for deletion. The blocks-to-be-deleted are sent to the datanode as a response to the next heartbeat RPC. The namenode sends only 100 blocks-to-be-deleted at a time. This was introduced as part of hadoop-994. The bug is that if the number of blocks-to-be-deleted exceeds 100, then that namenode marks all the remaining blocks in the block report for deletion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1135) A block report processing may
incorrectly cause the namenode to delete blocks
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483064 ]
Andrzej Bialecki commented on HADOOP-1135:
-------------------------------------------
IMHO this should go to a 0.12.2 release - this looks like pretty serious issue, and at the moment it's still well isolated from other changes in the trunk.
> A block report processing may incorrectly cause the namenode to delete blocks
> ------------------------------------------------------------------------------
>
> Key: HADOOP-1135
> URL: https://issues.apache.org/jira/browse/HADOOP-1135
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Fix For: 0.13.0
>
> Attachments: blockReportInvalidateBlock2.patch
>
>
> When a block report arrives at the namenode, the namenode goes through all the blocks on that datanode. If a block is not valid it is marked for deletion. The blocks-to-be-deleted are sent to the datanode as a response to the next heartbeat RPC. The namenode sends only 100 blocks-to-be-deleted at a time. This was introduced as part of hadoop-994. The bug is that if the number of blocks-to-be-deleted exceeds 100, then that namenode marks all the remaining blocks in the block report for deletion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1135) A block report processing may
incorrectly cause the namenode to delete blocks
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482513 ]
Hairong Kuang commented on HADOOP-1135:
---------------------------------------
+1. The logic looks correct. This makes sure that only invalid blocks will be deleted.
> A block report processing may incorrectly cause the namenode to delete blocks
> ------------------------------------------------------------------------------
>
> Key: HADOOP-1135
> URL: https://issues.apache.org/jira/browse/HADOOP-1135
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: blockReportInvalidateBlock2.patch
>
>
> When a block report arrives at the namenode, the namenode goes through all the blocks on that datanode. If a block is not valid it is marked for deletion. The blocks-to-be-deleted are sent to the datanode as a response to the next heartbeat RPC. The namenode sends only 100 blocks-to-be-deleted at a time. This was introduced as part of hadoop-994. The bug is that if the number of blocks-to-be-deleted exceeds 100, then that namenode marks all the remaining blocks in the block report for deletion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1135) A block report processing may
incorrectly cause the namenode to delete blocks
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-1135:
------------------------------
Resolution: Fixed
Fix Version/s: 0.13.0
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Dhruba!
(I've marked it as fixed in 0.13.0, but there is still an open question as to whether this merits a 0.12.2 release.)
> A block report processing may incorrectly cause the namenode to delete blocks
> ------------------------------------------------------------------------------
>
> Key: HADOOP-1135
> URL: https://issues.apache.org/jira/browse/HADOOP-1135
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Fix For: 0.13.0
>
> Attachments: blockReportInvalidateBlock2.patch
>
>
> When a block report arrives at the namenode, the namenode goes through all the blocks on that datanode. If a block is not valid it is marked for deletion. The blocks-to-be-deleted are sent to the datanode as a response to the next heartbeat RPC. The namenode sends only 100 blocks-to-be-deleted at a time. This was introduced as part of hadoop-994. The bug is that if the number of blocks-to-be-deleted exceeds 100, then that namenode marks all the remaining blocks in the block report for deletion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1135) A block report processing may
incorrectly cause the namenode to delete blocks
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1135:
-------------------------------------
Attachment: blockReportInvalidateBlock2.patch
Code uploaded for code review. Unit test coming soon.
> A block report processing may incorrectly cause the namenode to delete blocks
> ------------------------------------------------------------------------------
>
> Key: HADOOP-1135
> URL: https://issues.apache.org/jira/browse/HADOOP-1135
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: blockReportInvalidateBlock2.patch
>
>
> When a block report arrives at the namenode, the namenode goes through all the blocks on that datanode. If a block is not valid it is marked for deletion. The blocks-to-be-deleted are sent to the datanode as a response to the next heartbeat RPC. The namenode sends only 100 blocks-to-be-deleted at a time. This was introduced as part of hadoop-994. The bug is that if the number of blocks-to-be-deleted exceeds 100, then that namenode marks all the remaining blocks in the block report for deletion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1135) A block report processing may
incorrectly cause the namenode to delete blocks
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482898 ]
Doug Cutting commented on HADOOP-1135:
--------------------------------------
Does this warrant a 0.12.2 release? It sounds like it could cause data loss...
> A block report processing may incorrectly cause the namenode to delete blocks
> ------------------------------------------------------------------------------
>
> Key: HADOOP-1135
> URL: https://issues.apache.org/jira/browse/HADOOP-1135
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: blockReportInvalidateBlock2.patch
>
>
> When a block report arrives at the namenode, the namenode goes through all the blocks on that datanode. If a block is not valid it is marked for deletion. The blocks-to-be-deleted are sent to the datanode as a response to the next heartbeat RPC. The namenode sends only 100 blocks-to-be-deleted at a time. This was introduced as part of hadoop-994. The bug is that if the number of blocks-to-be-deleted exceeds 100, then that namenode marks all the remaining blocks in the block report for deletion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1135) A block report processing may
incorrectly cause the namenode to delete blocks
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-1135:
--------------------------------
Fix Version/s: (was: 0.13.0)
0.12.2
> A block report processing may incorrectly cause the namenode to delete blocks
> ------------------------------------------------------------------------------
>
> Key: HADOOP-1135
> URL: https://issues.apache.org/jira/browse/HADOOP-1135
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Fix For: 0.12.2
>
> Attachments: blockReportInvalidateBlock2.patch
>
>
> When a block report arrives at the namenode, the namenode goes through all the blocks on that datanode. If a block is not valid it is marked for deletion. The blocks-to-be-deleted are sent to the datanode as a response to the next heartbeat RPC. The namenode sends only 100 blocks-to-be-deleted at a time. This was introduced as part of hadoop-994. The bug is that if the number of blocks-to-be-deleted exceeds 100, then that namenode marks all the remaining blocks in the block report for deletion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1135) A block report processing may
incorrectly cause the namenode to delete blocks
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1135:
-------------------------------------
Summary: A block report processing may incorrectly cause the namenode to delete blocks (was: A block report processing may incorrect cause the namenode to delete blocks )
> A block report processing may incorrectly cause the namenode to delete blocks
> ------------------------------------------------------------------------------
>
> Key: HADOOP-1135
> URL: https://issues.apache.org/jira/browse/HADOOP-1135
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
>
> When a block report arrives at the namenode, the namenode goes through all the blocks on that datanode. If a block is not valid it is marked for deletion. The blocks-to-be-deleted are sent to the datanode as a response to the next heartbeat RPC. The namenode sends only 100 blocks-to-be-deleted at a time. This was introduced as part of hadoop-994. The bug is that if the number of blocks-to-be-deleted exceeds 100, then that namenode marks all the remaining blocks in the block report for deletion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1135) A block report processing may
incorrectly cause the namenode to delete blocks
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-1135:
-------------------------------------
Status: Patch Available (was: Open)
> A block report processing may incorrectly cause the namenode to delete blocks
> ------------------------------------------------------------------------------
>
> Key: HADOOP-1135
> URL: https://issues.apache.org/jira/browse/HADOOP-1135
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: blockReportInvalidateBlock2.patch
>
>
> When a block report arrives at the namenode, the namenode goes through all the blocks on that datanode. If a block is not valid it is marked for deletion. The blocks-to-be-deleted are sent to the datanode as a response to the next heartbeat RPC. The namenode sends only 100 blocks-to-be-deleted at a time. This was introduced as part of hadoop-994. The bug is that if the number of blocks-to-be-deleted exceeds 100, then that namenode marks all the remaining blocks in the block report for deletion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1135) A block report processing may
incorrectly cause the namenode to delete blocks
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12482955 ]
dhruba borthakur commented on HADOOP-1135:
------------------------------------------
I agree that this could cause data loss.
> A block report processing may incorrectly cause the namenode to delete blocks
> ------------------------------------------------------------------------------
>
> Key: HADOOP-1135
> URL: https://issues.apache.org/jira/browse/HADOOP-1135
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
> Assigned To: dhruba borthakur
> Attachments: blockReportInvalidateBlock2.patch
>
>
> When a block report arrives at the namenode, the namenode goes through all the blocks on that datanode. If a block is not valid it is marked for deletion. The blocks-to-be-deleted are sent to the datanode as a response to the next heartbeat RPC. The namenode sends only 100 blocks-to-be-deleted at a time. This was introduced as part of hadoop-994. The bug is that if the number of blocks-to-be-deleted exceeds 100, then that namenode marks all the remaining blocks in the block report for deletion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.