You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/08/03 23:11:26 UTC
[jira] [Created] (HDFS-2223) Untangle depencencies between NN
components
Untangle depencencies between NN components
-------------------------------------------
Key: HDFS-2223
URL: https://issues.apache.org/jira/browse/HDFS-2223
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099664#comment-13099664 ]
Hudson commented on HDFS-2223:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #857 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/857/])
HDFS-2223. Untangle depencencies between NN components. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.24.0
>
> Attachments: hdfs-2223-1.txt, hdfs-2223-10.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082023#comment-13082023 ]
Aaron T. Myers commented on HDFS-2223:
--------------------------------------
+1, patch looks good to me.
One question: It seems like it wouldn't be too tough to also remove the reference to {{FSNamesystem}} from {{FSDirectory}}. Why not do that here? Feel free to say this should be done in a separate JIRA.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2223:
------------------------------
Attachment: hdfs-2223-2.txt
A little further cleanup - no longer need to keep checkpointDirs around as state inside FSImage, since it's truly a load-time thing.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099662#comment-13099662 ]
Hudson commented on HDFS-2223:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #923 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/923/])
HDFS-2223. Untangle depencencies between NN components. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.24.0
>
> Attachments: hdfs-2223-1.txt, hdfs-2223-10.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2223:
---------------------------------
Attachment: hdfs-2223-3.txt
Exact same patch as Todd's, but generated with the --no-prefix option.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082568#comment-13082568 ]
Hadoop QA commented on HDFS-2223:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12489921/hdfs-2223-3.txt
against trunk revision 1155998.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1063//console
This message is automatically generated.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092980#comment-13092980 ]
Suresh Srinivas commented on HDFS-2223:
---------------------------------------
Comments:
# FSImage creating FSNamesystem() does not seem right. Especially when FSNamesystem#loadFSImage() calls FSImage#format(), which then creates a new FSNamesystem. It would be good to create FSNamsystem and pass it to FSImage in #format().
# BackupImage#journal() BackupImage#convergeJournalSpool() could take FSNamesystem as additional argument. We could remove FSNamesystem member variable. BTW I do not see any call to BackupImage#saveCheckPoint(). These changes can be done in a separate jira.
# Question unrelated to this patch: why is the expectation that configuration have both DFS_NAMENODE_CHECKPOINT_EDITS_DIR_KEY and DFS_NAMENODE_CHECKPOINT_DIR_KEY set, in FSImage#doImportCheckpoint()?
# Please add javadoc to FSNamesystem#loadFromDisk()
# Minor: FSNamesystem#initialize() - systemStart variable is no longer used
# Minor: Unnecessary change in NNStorage.java that pushes code beyond 80 columns
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2223:
------------------------------
Attachment: hdfs-2223-9.txt
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097321#comment-13097321 ]
Hadoop QA commented on HDFS-2223:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12492545/hdfs-2223-7.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 18 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1196//console
This message is automatically generated.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2223:
------------------------------
Status: Open (was: Patch Available)
Looks like at least one of those test issues is legit. Looking into it.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091926#comment-13091926 ]
Todd Lipcon commented on HDFS-2223:
-----------------------------------
I ran TestHost2NodesMap locally and it passed - it's been failing on hudson lately, tracked by a different JIRA.
Suresh, can you give a +1 if the patch looks good? Thanks!
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097646#comment-13097646 ]
Suresh Srinivas commented on HDFS-2223:
---------------------------------------
+1 for the latest patch with -1s from hudson addressed.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097709#comment-13097709 ]
Hadoop QA commented on HDFS-2223:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493087/hdfs-2223-9.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 18 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
-1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestDfsOverAvroRpc
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1200//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1200//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1200//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1200//console
This message is automatically generated.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082082#comment-13082082 ]
Suresh Srinivas commented on HDFS-2223:
---------------------------------------
bq. container for BlockManager, DatanodeManager, DelegationTokens, LeaseManager, etc
Why not container for FSDirectory, FSImage, FSEditlog, as they are very closely related to namespace.
Should NNStorage belong to NameNode?
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099630#comment-13099630 ]
Hadoop QA commented on HDFS-2223:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493506/hdfs-2223-10.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 18 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestDfsOverAvroRpc
org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap
org.apache.hadoop.hdfs.server.datanode.TestReplicasMap
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1222//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1222//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1222//console
This message is automatically generated.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-10.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2223:
------------------------------
Attachment: hdfs-2223-1.txt
Here's a patch which does the following:
- FSImage no longer has a reference to FSNamesystem. Instead, the FSNamesystem is passed in as a parameter to save/load.
- FSDirectory no longer is responsible for creating FSImage, and no longer goes through FSImage to get FSNamesystem. I wasn't able to completely remove the {{fsImage}} member in FSDirectory yet, but that's the next logical step.
- FSDirectory no longer coordinates the loading of the image at startup. This code is in a new static method {{FSNamesystem.loadFromDisk}} which makes a little more sense
- Removed some constructors from FSNamesystem that were made extraneous by the above work
The tests I've run so far seem to be passing.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099302#comment-13099302 ]
Todd Lipcon commented on HDFS-2223:
-----------------------------------
How about this: we can keep getNamespaceInfo with the readlock, and delegate inside the lock to getNamespaceInfoInternal(). From within saveNamespace, we can directly call the getNamespaceInfoInternal method, since we know that the outer coordinating thread of saveNamespace is holding the lock "on our behalf".
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2223:
------------------------------
Attachment: hdfs-2223-8.txt
Rebased on trunk
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097647#comment-13097647 ]
Todd Lipcon commented on HDFS-2223:
-----------------------------------
Looking into the -1s, I see that this is causing TestEditLogRace.testSaveNamespace to time out. It's a kind of messy situation -- the FSN rwlock is deadlocking against its fairness policy:
- saveNamespace acquires the read lock
- spawns an image saver thread
- another thread comes in to do mkdirs and is waiting on the write lock
- the image saver thread calls getNamespaceInfo (new in this patch) and wants the read lock
Fairness policy says that the image saver thread can't get the read lock, since someone is already waiting on the write lock. So, it hangs there. The writer hangs on the main thread, and the main thread is join()ing on the saver thread.
I think the easiest solution is to unsynchronize getNamespaceInfo, since in practice none of those fields change after the FSN is initialized. I will upload a new patch.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094866#comment-13094866 ]
Todd Lipcon commented on HDFS-2223:
-----------------------------------
Thanks for the comments, Suresh. I will work on a new patch.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2223:
------------------------------
Attachment: hdfs-2223-10.txt
Here's a new version of the patch, rebased on trunk, implementing the above idea.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-10.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091876#comment-13091876 ]
Hadoop QA commented on HDFS-2223:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12491802/hdfs-2223-6.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 18 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1160//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1160//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1160//console
This message is automatically generated.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2223:
------------------------------
Resolution: Fixed
Fix Version/s: 0.24.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to trunk for now. We may want to merge this to 0.23 and/or HA soon as well, but will hold off on that for the time being. Thanks for reviewing, Suresh.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.24.0
>
> Attachments: hdfs-2223-1.txt, hdfs-2223-10.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2223:
------------------------------
Attachment: hdfs-2223-4.txt
Rebased on trunk
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099095#comment-13099095 ]
Suresh Srinivas commented on HDFS-2223:
---------------------------------------
Thinking a bit more - cluster id, block pool ID etc. also will have unsynchronized access even with my suggestion.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100276#comment-13100276 ]
Hudson commented on HDFS-2223:
------------------------------
Integrated in Hadoop-Hdfs-trunk #787 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/787/])
HDFS-2223. Untangle depencencies between NN components. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.24.0
>
> Attachments: hdfs-2223-1.txt, hdfs-2223-10.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2223:
---------------------------------
Status: Patch Available (was: Open)
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2223:
------------------------------
Attachment: hdfs-2223-5.txt
Diffs since previous patch:
- only register the FSN MBean when the NN is activated, not when it's constructed
- sets storage.upgradeManager earlier during startup to fix TestDistributedUpgrade
- improves TestDistributedUpgrade to be easier to debug
- fixes javadoc warnings (trivial)
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082043#comment-13082043 ]
Tsz Wo (Nicholas), SZE commented on HDFS-2223:
----------------------------------------------
> FSImage no longer has a reference to FSNamesystem. Instead, the FSNamesystem is passed in as a parameter to save/load.
Should FSImage.namesystem be removed?
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079034#comment-13079034 ]
Todd Lipcon commented on HDFS-2223:
-----------------------------------
First, here's my conception of what the responsibilities and interactions are meant to be:
*NameNode*: container class that handles startup/shutdown of the Java process itself. Starts/stops RPC and HTTP interfaces.
*FSNamesystem*: container of both transient and persisted namespace state:
-- container for BlockManager, DatanodeManager, DelegationTokens, LeaseManager, etc
-- RPC calls that modify or inspect the namespace should get delegated here
--- anything that touches only blocks (eg block reports) should get delegated to BlockManager
--- anything that touches only file information (eg permissions, mkdirs) should get delegated to FSDirectory
--- anything that crosses two of the above components should be coordinated here
-- should log mutations to FSEditLog
*FSDirectory*:
-- Namespace state that persists across restarts: the tree of inodes, and for each inode, the block list
-- should log mutations to FSEditLog
*FSImage*: responsible for the mechanisms by which an FSNamesystem is snapshotted and logged
-- should be responsible for lifecycle of FSEditLog and NNStorage
*NNStorage*: should only be responsible for file management inside storage directories, and tracking failures states of those directories
-- should not have any references "up" to other classes
*FSEditLog*: responsible for managing a set of journals
-- after initialization, should not need to reference any other classes
Hopefully this jives with what other people understand to be the "ideal" we're going for.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082826#comment-13082826 ]
Hadoop QA commented on HDFS-2223:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12490055/hdfs-2223-4.txt
against trunk revision 1155998.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.server.blockmanagement.TestReplicationPolicy
org.apache.hadoop.hdfs.server.common.TestDistributedUpgrade
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings
org.apache.hadoop.hdfs.TestHDFSServerPorts
org.apache.hadoop.tools.TestJMXGet
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1079//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1079//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1079//console
This message is automatically generated.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099661#comment-13099661 ]
Hudson commented on HDFS-2223:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #846 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/846/])
HDFS-2223. Untangle depencencies between NN components. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.24.0
>
> Attachments: hdfs-2223-1.txt, hdfs-2223-10.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098495#comment-13098495 ]
Todd Lipcon commented on HDFS-2223:
-----------------------------------
The above tests and release audit warnings are known issues in trunk addressed by other JIRAs. Suresh, if the above comment about getNamespaceInfo synchronization makes sense, I will go ahead and commit.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Tsz Wo (Nicholas), SZE (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-2223:
-----------------------------------------
Fix Version/s: 0.23.3
I have merged this to 0.23.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.24.0, 0.23.3
>
> Attachments: hdfs-2223-1.txt, hdfs-2223-10.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100295#comment-13100295 ]
Hudson commented on HDFS-2223:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #810 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/810/])
HDFS-2223. Untangle depencencies between NN components. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166466
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.24.0
>
> Attachments: hdfs-2223-1.txt, hdfs-2223-10.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2223:
------------------------------
Status: Patch Available (was: Open)
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098634#comment-13098634 ]
Suresh Srinivas commented on HDFS-2223:
---------------------------------------
I have some concerns about removing readLock() completely. While we do not expect the values to change, the visibility is not guaranteed without locks. How about the following changes:
# upgradeManager.getUpgradeVersion() is a synchronized method. So it does not required readLock()
# FSNamesystem#dir is non volatile - a thread accessing it could just see null stale value. Could you make it final and initialize it in the constructor. BTW I have also been thinking that instead of using initialize method, which prevents making some member vars non final, we should initialize them in the constructor it self.
# Can you please make the following final FSImage#storage, FSDirectory#fsImage
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-2223:
----------------------------------
Attachment: hdfs-2223-6.txt
Rebased patch to work with new trunk.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082771#comment-13082771 ]
Todd Lipcon commented on HDFS-2223:
-----------------------------------
bq. Should FSImage.namesystem be removed?
Yes - it is removed in this patch. I wasn't able to easily remove it from BackupImage at this point, without changing a bunch of the BN around. So, I left it in the subclass only.
bq. Why not container for FSDirectory, FSImage, FSEditlog, as they are very closely related to namespace.
bq. Should NNStorage belong to NameNode?
I think FSEditLog still belongs below FSImage, as it's part of the "image on disk" in a sense, and their lifecycle is tightly related.
Moving NNStorage up into NameNode does make a certain amount of sense to me, but I didn't want to bite off too much in one patch.
Do you think the changes in the attached patch are good, even if they're not a complete cleanup of everything? If so, let's commit this and then do some more work in another JIRA, just so each small improvement is easy to review?
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099650#comment-13099650 ]
Suresh Srinivas commented on HDFS-2223:
---------------------------------------
bq. How about this: we can keep getNamespaceInfo with the readlock, and delegate inside the lock to getNamespaceInfoInternal(). From within saveNamespace, we can directly call the getNamespaceInfoInternal method, since we know that the outer coordinating thread of saveNamespace is holding the lock "on our behalf".
Nice idea. +1 for the patch.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-10.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2223:
------------------------------
Attachment: hdfs-2223-7.txt
bq. FSImage creating FSNamesystem() does not seem right. Especially when FSNamesystem#loadFSImage() calls FSImage#format(), which then creates a new FSNamesystem. It would be good to create FSNamsystem and pass it to FSImage in #format().
Fixed. I had to also move the initialization of NNResourceChecker, since with this modification FSNamesystem was being created before FSImage. NNResourceChecker should probably move down into FSImage at some point, but I didn't want to broaden the scope of this JIRA further.
bq. BackupImage#journal() BackupImage#convergeJournalSpool() could take FSNamesystem as additional argument. We could remove FSNamesystem member variable. BTW I do not see any call to BackupImage#saveCheckPoint(). These changes can be done in a separate jira.
Agreed this could be done separately.
bq. Question unrelated to this patch: why is the expectation that configuration have both DFS_NAMENODE_CHECKPOINT_EDITS_DIR_KEY and DFS_NAMENODE_CHECKPOINT_DIR_KEY set, in FSImage#doImportCheckpoint()?
The {{dfs.namenode.checkpoint.edits.dir}} configuration defaults to {{${dfs.namenode.checkpoint.dir}}}, so setting just the former without explicitly setting the latter is sufficient.
bq. Please add javadoc to FSNamesystem#loadFromDisk()
Done
bq. Minor: FSNamesystem#initialize() - systemStart variable is no longer used
It's still used from getStartTime() and SafeMode.leave.
bq. Minor: Unnecessary change in NNStorage.java that pushes code beyond 80 columns
Oops, fixed.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN
components
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097615#comment-13097615 ]
Hadoop QA commented on HDFS-2223:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493075/hdfs-2223-8.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 18 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
-1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1197//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1197//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1197//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1197//console
This message is automatically generated.
> Untangle depencencies between NN components
> -------------------------------------------
>
> Key: HDFS-2223
> URL: https://issues.apache.org/jira/browse/HDFS-2223
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2223-1.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt
>
>
> Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira