You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2011/12/16 00:19:30 UTC
[jira] [Created] (HDFS-2693) Synchronization issues around state
transition
Synchronization issues around state transition
----------------------------------------------
Key: HDFS-2693
URL: https://issues.apache.org/jira/browse/HDFS-2693
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Priority: Critical
Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
(1) leave old state
(2) change state variable
(3) enter new state
This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2693) Synchronization issues around state
transition
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174068#comment-13174068 ]
Hudson commented on HDFS-2693:
------------------------------
Integrated in Hadoop-Hdfs-HAbranch-build #23 (See [https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/23/])
HDFS-2693. Fix synchronization issues around state transition. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1221582
Files :
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-1623.txt
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/HAContext.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/HAState.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/StandbyState.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCorruption.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBackupNode.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestEditLogTailer.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestEditLogsDuringFailover.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAStateTransitions.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyIsHot.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt
>
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2693) Synchronization issues around state
transition
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2693:
------------------------------
Attachment: hdfs-2693.txt
New patch addresses the javadoc nit
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt
>
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2693) Synchronization issues around state
transition
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173766#comment-13173766 ]
Todd Lipcon commented on HDFS-2693:
-----------------------------------
bq. Can't we nuke the new code in commitBlockSynchronization since we should never execute this, ie we don't need a TODO here right?
I was only moving it in this patch, so I didn't want to make a call on whether it's needed. I just noticed, when moving it, that it would never run. So I wanted to leave the TODO as a reminder that we need to look at pipeline recovery more closely, and here's one area where we clearly haven't handled it quite right (or need test coverage)
bq. Nit: HAContext needs javadocs for new methods.
Fixed, uploading new patch momentarily
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt
>
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2693) Synchronization issues around state
transition
Posted by "Todd Lipcon (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon resolved HDFS-2693.
-------------------------------
Resolution: Fixed
Fix Version/s: HA branch (HDFS-1623)
Hadoop Flags: Reviewed
Committed to branch, thx for the reviews Eli.
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt
>
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2693) Synchronization issues around state
transition
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2693:
------------------------------
Attachment: hdfs-2693.txt
One last rev to fix some other non-HA unit tests which were calling directly into BlockManager without locking the FSN.
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt
>
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2693) Synchronization issues around state
transition
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2693:
------------------------------
Attachment: hdfs-2693.txt
Attached patch fixes the bug mentioned above. I also had a similar bug in endCheckpoint.
In the review, please double-check that I didn't make the mistake anywhere else? (having the checkOperation outside the try...finally)
Thanks!
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2693.txt, hdfs-2693.txt
>
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2693) Synchronization issues around state
transition
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173783#comment-13173783 ]
Eli Collins commented on HDFS-2693:
-----------------------------------
bq. So I wanted to leave the TODO as a reminder that we need to look at pipeline recovery more closely
Makes sense.
+1 to the latest patch
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt
>
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2693) Synchronization issues around state
transition
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2693:
------------------------------
Attachment: hdfs-2693.txt
arg, accidentally generated patch from dirty working tree. take 2
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt
>
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2693) Synchronization issues around state
transition
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173731#comment-13173731 ]
Eli Collins commented on HDFS-2693:
-----------------------------------
Can't we nuke the new code in commitBlockSynchronization since we should never execute this, ie we don't need a TODO here right?
Nit: HAContext needs javadocs for new methods.
Otherwise looks great. Didn't see any synchronization bugs.
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt, hdfs-2693.txt
>
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2693) Synchronization issues around state
transition
Posted by "Todd Lipcon (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon reassigned HDFS-2693:
---------------------------------
Assignee: Todd Lipcon
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2693) Synchronization issues around state
transition
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172886#comment-13172886 ]
Todd Lipcon commented on HDFS-2693:
-----------------------------------
Found a nasty bug in the current patch: in {{getBlockLocationsUpdateTimes}} the {{checkOperation}} call has to move down inside {{try..catch}} or else calling this function on an NN in standby state will leak the FSN readlock, killing the SBN. I'll upload a new patch soon. Still worth reviewing the general patch since most of it seems to be working correctly.
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2693.txt
>
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2693) Synchronization issues around state
transition
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2693:
------------------------------
Attachment: hdfs-2693.txt
Patch implements the above described solution and includes a multi-threaded unit test that fails on the unpatched code.
I had to do a little bit of rejiggering of the HAContext interface to provide access to the lock, and also added a function for use in tests to allow us to read from the standby. We used to go through the FSN directly to avoid the "checkOperation" checks, but now that "checkOperation" is inside FSN, we needed another way around them. Eventually this could become a user-configurable option, but for now it's only meant for testing.
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-2693.txt
>
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2693) Synchronization issues around state
transition
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170571#comment-13170571 ]
Todd Lipcon commented on HDFS-2693:
-----------------------------------
This is a bit tricky -- I think we need to do two things:
a) take the FSN write lock around the full state transition
b) move all of the {{nn.checkOperation()}} calls inside the FSN lock
Otherwise we have a race where the {{checkOperation}} call happens outside the lock, then a state transition happens, then the operation can proceed even though it shouldn't be allowed.
> Synchronization issues around state transition
> ----------------------------------------------
>
> Key: HDFS-2693
> URL: https://issues.apache.org/jira/browse/HDFS-2693
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Priority: Critical
>
> Currently when the NN changes state, it does so without synchronization. In particular, the state transition function does:
> (1) leave old state
> (2) change state variable
> (3) enter new state
> This means that the NN is marked as "active" before it has actually transitioned to active mode and opened its edit logs. This gives a window where write transactions can come in and the {{checkOperation}} allows them, but then they fail because the edit log is not yet opened.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira