You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2009/08/18 01:18:14 UTC
[jira] Created: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
------------------------------------------------------------------------------------------------
Key: HDFS-550
URL: https://issues.apache.org/jira/browse/HDFS-550
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: 0.21.0
Reporter: Hairong Kuang
Assignee: Hairong Kuang
Priority: Blocker
Fix For: Append Branch
Current trunks calls first unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
There are two bugs with this implementation:
1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id coexist in a datanode. Also if the replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753237#action_12753237 ]
Hairong Kuang commented on HDFS-550:
------------------------------------
> The recovery code (at datanode restart time) first checks to see if the block is already in current.
The problem is that replicas are not in "current" normally. They are in the subdirs of "current".
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754403#action_12754403 ]
dhruba borthakur commented on HDFS-550:
---------------------------------------
Hi hairong, sorry for not getting back to you earlier. I like the option 2 that you have implemented in this patch. Option 1 is not safe to do on Windows platform.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12752918#action_12752918 ]
Hairong Kuang commented on HDFS-550:
------------------------------------
The problem is that when DataNode recovers detached blocks, it assumes that the original block is in the current directory which is most of the time not true in a production cluster. So the recovery adds duplicate replicas of the same block id to a datanode disk.
Maybe what you claimed is true, but the current trunk does not treat the temporary detached blocks as you explained. If what you claim is true, at least DataNode should treat the detached directory as a tmp directory and removes its content at Startup time.
My unit tests test only the new recovery works or not. So it may not serve your purpose.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang resolved HDFS-550.
--------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
I've just committed this.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch, detach1.patch, detach2.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744286#action_12744286 ]
Hairong Kuang commented on HDFS-550:
------------------------------------
New proposal:
1. when unlinking a block/meta file, the tempoary file stays in the same directory where the block/meta file is located. The temporary file has a suffix ".unlink". The guarantees that ".unlink" files stay in a snapshot when upgrading a datanode. It also keeps the location of the replica to be unlinked.
2. When datanodes restart, for a file with ".unlink" suffix, check if the original replica/meta exists or not. If exists, delete the ".unlink" file. If no, rename ".unlink" to be the original block/meta file name.
Here is the proposal to handle existing "detach" directory:
1. Datanodes refuses to startup if "detach" directory is not empty;
2. If "detach" directory exists and is empty, datanodes removes the directory during upgrade.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
>
> Current trunks calls first unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id coexist in a datanode. Also if the replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753134#action_12753134 ]
dhruba borthakur commented on HDFS-550:
---------------------------------------
> So the recovery adds duplicate replicas of the same block id to a datanode disk ....... (A)
This is true. This is harmless because they will get deleted in the next block report, isn't it?
> at least DataNode should treat the detached directory as a tmp directory and removes its content at Startup time...... (B)
I completely agree. The reason this is being done is because the FileUtil.renameFile() is not atomic on Windows. Hence it is better to have issue A (as marked above) rather than losing blocks on Windows platform.
> My unit tests test only the new recovery works or not.
I am fine with the code restructuring that you have done. +1. However, if this is supposed to fix a bug in the current code, isn't it a good idea to first write a unit test to trigger that problem and then demonstrate that the unit test works well with the code in the patch.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753233#action_12753233 ]
dhruba borthakur commented on HDFS-550:
---------------------------------------
The recovery code (at datanode restart time) first checks to see if the block is already in current. If not, it adds it to current. So, there can be only replica of that block in the "current" directory of the namenode.
Another copy of the same block could also reside in the "snapshot" directory. Is that what you mean by "duplicate replicas on the same datanode"?
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760160#action_12760160 ]
dhruba borthakur commented on HDFS-550:
---------------------------------------
+1 code looks good. From my understanding, the effective change done by this patch is to keep the detached file in the same directory as the original block, right?
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12753203#action_12753203 ]
Hairong Kuang commented on HDFS-550:
------------------------------------
> This is harmless because they will get deleted in the next block report, isn't it?
No, this is not harmless. That's the key point. Block report does not handle duplicate replicas from one datanode. More problematic is that the temporary replicas under "detached" may be corrupt if DataNode dies when copy is in progress. So recovery introduces corrupt replicas.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748056#action_12748056 ]
Konstantin Shvachko commented on HDFS-550:
------------------------------------------
Suffix ".unlinked" would be better.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760292#action_12760292 ]
Hairong Kuang commented on HDFS-550:
------------------------------------
Dhruba, thanks a lot for your review!
> the effective change done by this patch is to keep the detached file in the same directory as the original block, right?
Yes, you are right. This is a simple strategy but fixes the two bugs described in the jira.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751729#action_12751729 ]
dhruba borthakur commented on HDFS-550:
---------------------------------------
The current code in the trunk actually copies the block file to the detach directory and then renames the newly created file from the detach directory to the original location of the block. The recover code that runs on datanode will actually always find that the files in the detach directory always exist in the original volume and will simply delete it from the detach directory. So, my guess is that the two bugs suggested in this JIRA does not exist. (The caveat is that FileUtil.renameFile() is atomic on Linux but not atomic on windows, hence the existence of the detach dir recovery code)
If we stlll want to go ahead with this JIRA, can you pl explain why keeping the detached file in the same directory helps?
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-550:
-------------------------------
Attachment: detach.patch
This patch implements the above logic and changes all "detach" methods to be "unlink" methods for the readability purpose.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-550:
-------------------------------
Attachment: detach2.patch
Unit tests were passed except for the known ones.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch, detach1.patch, detach2.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-550:
-------------------------------
Description:
Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
There are two bugs with this implementation:
1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
was:
Current trunks calls first unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
There are two bugs with this implementation:
1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id coexist in a datanode. Also if the replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-550:
-------------------------------
Issue Type: Sub-task (was: Bug)
Parent: HDFS-265
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HDFS-550:
-------------------------------
Attachment: detach1.patch
A updated patch that merges with the append branch.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch, detach1.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760355#action_12760355 ]
Hairong Kuang commented on HDFS-550:
------------------------------------
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 7 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch, detach1.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754214#action_12754214 ]
Hairong Kuang commented on HDFS-550:
------------------------------------
Dhruba, does my last comment make sense? I see we have two options for handling temporary detached blocks at DataNode startup time:
1. Do not recover them. I would propose to put the temporary detached blocks in the "tmp" directory so they get automatically deleted when DataNode starts.
2. Recover them. Then the temporary .detach files have to be in a snapshot and need a way to figure out which directory the original files are located. I think the proposal I put in this jira should work.
Let me know which option you prefer.
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-550) DataNode restarts may introduce
corrupt/duplicated/lost replicas when handling detached replicas
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751731#action_12751731 ]
dhruba borthakur commented on HDFS-550:
---------------------------------------
Since you have a patch that contains unit test cases, can I run the unit test on the current trunk to see if it encounters the problems you have listed?
> DataNode restarts may introduce corrupt/duplicated/lost replicas when handling detached replicas
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-550
> URL: https://issues.apache.org/jira/browse/HDFS-550
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Priority: Blocker
> Fix For: Append Branch
>
> Attachments: detach.patch
>
>
> Current trunk first calls detach to unlinks a finalized replica before appending to this block. Unlink is done by temporally copying the block file in the "current" subtree to a directory called "detach" under the volume's daa directory and then copies it back when unlink succeeds. On datanode restarts, datanodes recover faied unlink by copying replicas under "detach" to "current".
> There are two bugs with this implementation:
> 1. The "detach" directory does not include in a snapshot. so rollback will cause the "detaching" replicas to be lost.
> 2. After a replica is copied to the "detach" directory, the information of its original location is lost. The current implementation erroneously assumes that the replica to be unlinked is under "current". This will make two instances of replicas with the same block id to coexist in a datanode. Also if a replica under "detach" is corrupt, the corrupt replica is moved to "current" without being detected, polluting datanode data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.