You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/05/25 03:18:47 UTC
[jira] [Created] (HDFS-1994) Fix race conditions when running two
rapidly checkpointing 2NNs
Fix race conditions when running two rapidly checkpointing 2NNs
---------------------------------------------------------------
Key: HDFS-1994
URL: https://issues.apache.org/jira/browse/HDFS-1994
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Todd Lipcon
Assignee: Todd Lipcon
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1994) Fix race conditions when running two
rapidly checkpointing 2NNs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038909#comment-13038909 ]
Todd Lipcon commented on HDFS-1994:
-----------------------------------
issue #1 is fairly straightforward to address - just need to write to a side file and rename into place
the issue #2 is a bit subtle. The interleaving is the following:
2NN A) calls rollEdits()
2NN B) calls rollEdits()
2NN A) calls getRemoteEditLogManifest()
2NN B) calls getRemoteEditLogManifest()
both of them then see the same manifest, and download all of the edits available. Therefore they merge to the same transaction.
This can be fixed one of two ways:
a) add a lock to the code that downloads an image, to make sure that only one image with a certain txid could be downloaded at a time
b) when the 2NN calls rollEdits, it should only try to checkpoint up to the transaction where the roll happened.
IMO we should actually do both: (a) as a sanity check, and (b) to ensure each checkpoint is unique, thus allowing us freedom later to do something like include the 2NN's hostname in the image itself
> Fix race conditions when running two rapidly checkpointing 2NNs
> ---------------------------------------------------------------
>
> Key: HDFS-1994
> URL: https://issues.apache.org/jira/browse/HDFS-1994
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-1994.txt
>
>
> HDFS-1984 added the ability to run two secondary namenodes at the same time. However, there were two races I found when stress testing this (by running two NNs each checkpointing in a tight loop with no sleep):
> 1) the writing of the seen_txid file was not atomic, so it was at some points reading an empty file
> 2) it was possible for two checkpointers to try to take a checkpoint at the same transaction ID, which would cause the two image downloads to collide and fail
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1994) Fix race conditions when running two
rapidly checkpointing 2NNs
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040409#comment-13040409 ]
Eli Collins commented on HDFS-1994:
-----------------------------------
+1 looks great.
DelegateAnswer seems generally useful, I'm surprised Mockito doesn't have this class itself.
Nits:
- s/uploader/uploaded/
- Remove the System.err.println("AHHHHHHHHHHHHHHHh") in AFOS
> Fix race conditions when running two rapidly checkpointing 2NNs
> ---------------------------------------------------------------
>
> Key: HDFS-1994
> URL: https://issues.apache.org/jira/browse/HDFS-1994
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-1994.txt, hdfs-1994.txt
>
>
> HDFS-1984 added the ability to run two secondary namenodes at the same time. However, there were two races I found when stress testing this (by running two NNs each checkpointing in a tight loop with no sleep):
> 1) the writing of the seen_txid file was not atomic, so it was at some points reading an empty file
> 2) it was possible for two checkpointers to try to take a checkpoint at the same transaction ID, which would cause the two image downloads to collide and fail
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1994) Fix race conditions when running two
rapidly checkpointing 2NNs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-1994:
------------------------------
Component/s: name-node
Description:
HDFS-1984 added the ability to run two secondary namenodes at the same time. However, there were two races I found when stress testing this (by running two NNs each checkpointing in a tight loop with no sleep):
1) the writing of the seen_txid file was not atomic, so it was at some points reading an empty file
2) it was possible for two checkpointers to try to take a checkpoint at the same transaction ID, which would cause the two image downloads to collide and fail
Affects Version/s: Edit log branch (HDFS-1073)
Fix Version/s: Edit log branch (HDFS-1073)
> Fix race conditions when running two rapidly checkpointing 2NNs
> ---------------------------------------------------------------
>
> Key: HDFS-1994
> URL: https://issues.apache.org/jira/browse/HDFS-1994
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
>
> HDFS-1984 added the ability to run two secondary namenodes at the same time. However, there were two races I found when stress testing this (by running two NNs each checkpointing in a tight loop with no sleep):
> 1) the writing of the seen_txid file was not atomic, so it was at some points reading an empty file
> 2) it was possible for two checkpointers to try to take a checkpoint at the same transaction ID, which would cause the two image downloads to collide and fail
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1994) Fix race conditions when running two
rapidly checkpointing 2NNs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-1994:
------------------------------
Attachment: hdfs-1994.txt
Updated patch that does solution "a" above. Also includes a new test case to trigger the interleaving and make sure only one is successful.
If we decide we also want to do "b" above, it could be done in a followup. But it's a very rare race that only really shows up in this kind of stress test, and it doesn't cause any corruption, just a missed checkpoint.
> Fix race conditions when running two rapidly checkpointing 2NNs
> ---------------------------------------------------------------
>
> Key: HDFS-1994
> URL: https://issues.apache.org/jira/browse/HDFS-1994
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-1994.txt, hdfs-1994.txt
>
>
> HDFS-1984 added the ability to run two secondary namenodes at the same time. However, there were two races I found when stress testing this (by running two NNs each checkpointing in a tight loop with no sleep):
> 1) the writing of the seen_txid file was not atomic, so it was at some points reading an empty file
> 2) it was possible for two checkpointers to try to take a checkpoint at the same transaction ID, which would cause the two image downloads to collide and fail
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1994) Fix race conditions when running two
rapidly checkpointing 2NNs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-1994:
------------------------------
Attachment: hdfs-1994.txt
Here's a preliminary patch. It needs some more unit tests that trigger the interleaving described above.
Doing the stress test described, it now runs a lot more stably, though there's still one unrelated issue in the 2NN (to be addressed later)
> Fix race conditions when running two rapidly checkpointing 2NNs
> ---------------------------------------------------------------
>
> Key: HDFS-1994
> URL: https://issues.apache.org/jira/browse/HDFS-1994
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-1994.txt
>
>
> HDFS-1984 added the ability to run two secondary namenodes at the same time. However, there were two races I found when stress testing this (by running two NNs each checkpointing in a tight loop with no sleep):
> 1) the writing of the seen_txid file was not atomic, so it was at some points reading an empty file
> 2) it was possible for two checkpointers to try to take a checkpoint at the same transaction ID, which would cause the two image downloads to collide and fail
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1994) Fix race conditions when running two
rapidly checkpointing 2NNs
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon resolved HDFS-1994.
-------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Fixed the nits and committed to branch.
Unfortunately Mockito doesn't provide that Answer implementation, but I based that implementation on a suggestion I found on the Mockito bugtracker.
> Fix race conditions when running two rapidly checkpointing 2NNs
> ---------------------------------------------------------------
>
> Key: HDFS-1994
> URL: https://issues.apache.org/jira/browse/HDFS-1994
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
> Affects Versions: Edit log branch (HDFS-1073)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: Edit log branch (HDFS-1073)
>
> Attachments: hdfs-1994.txt, hdfs-1994.txt
>
>
> HDFS-1984 added the ability to run two secondary namenodes at the same time. However, there were two races I found when stress testing this (by running two NNs each checkpointing in a tight loop with no sleep):
> 1) the writing of the seen_txid file was not atomic, so it was at some points reading an empty file
> 2) it was possible for two checkpointers to try to take a checkpoint at the same transaction ID, which would cause the two image downloads to collide and fail
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira