You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2011/11/24 05:35:40 UTC
[jira] [Created] (HDFS-2591) HA: MiniDFSCluster support to mix and
match federation with HA
HA: MiniDFSCluster support to mix and match federation with HA
--------------------------------------------------------------
Key: HDFS-2591
URL: https://issues.apache.org/jira/browse/HDFS-2591
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha, test
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Right now the MiniDFS builder is somewhat inflexible - it just takes a "numNameNodes" parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs -- eg to test the case where there is one nameservice that is HA and another which is not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2591) HA: MiniDFSCluster support to mix
and match federation with HA
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159100#comment-13159100 ]
Eli Collins commented on HDFS-2591:
-----------------------------------
+1 lgtm
> HA: MiniDFSCluster support to mix and match federation with HA
> --------------------------------------------------------------
>
> Key: HDFS-2591
> URL: https://issues.apache.org/jira/browse/HDFS-2591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2591-v2.txt, hdfs-2591.txt
>
>
> Right now the MiniDFS builder is somewhat inflexible - it just takes a "numNameNodes" parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs -- eg to test the case where there is one nameservice that is HA and another which is not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2591) HA: MiniDFSCluster support to mix
and match federation with HA
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158024#comment-13158024 ]
Eli Collins commented on HDFS-2591:
-----------------------------------
What you have so far looks good. If we end up creating a class for the map in HDFS-2582 it could probably share the code here since it's mostly the same map essentially in class form.
> HA: MiniDFSCluster support to mix and match federation with HA
> --------------------------------------------------------------
>
> Key: HDFS-2591
> URL: https://issues.apache.org/jira/browse/HDFS-2591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2591.txt
>
>
> Right now the MiniDFS builder is somewhat inflexible - it just takes a "numNameNodes" parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs -- eg to test the case where there is one nameservice that is HA and another which is not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2591) HA: MiniDFSCluster support to mix and
match federation with HA
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2591:
------------------------------
Attachment: hdfs-2591-v3.txt
Small update to fix some unit test failures discovered when testing this patch against the tip of branch:
- add a toString method to ConfiguredNNAddress (helpful when debugging)
- change {{addNameNode}} to fail when trying to add a NN to a minicluster which wasn't set up as federated (to fix failure of TestDatanodeMultipleRegistrations)
- fix TestDatanodeMulitipleBlockScanner to use the DN's old configuration when refreshing it
Fairly trivial delta between the two, so I'll commit unless there's any comment in the next few hours.
> HA: MiniDFSCluster support to mix and match federation with HA
> --------------------------------------------------------------
>
> Key: HDFS-2591
> URL: https://issues.apache.org/jira/browse/HDFS-2591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2591-v2.txt, hdfs-2591-v3.txt, hdfs-2591.txt
>
>
> Right now the MiniDFS builder is somewhat inflexible - it just takes a "numNameNodes" parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs -- eg to test the case where there is one nameservice that is HA and another which is not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2591) HA: MiniDFSCluster support to mix and
match federation with HA
Posted by "Todd Lipcon (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon resolved HDFS-2591.
-------------------------------
Resolution: Fixed
Fix Version/s: HA branch (HDFS-1623)
Hadoop Flags: Reviewed
Committed v3 to HA branch
> HA: MiniDFSCluster support to mix and match federation with HA
> --------------------------------------------------------------
>
> Key: HDFS-2591
> URL: https://issues.apache.org/jira/browse/HDFS-2591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2591-v2.txt, hdfs-2591-v3.txt, hdfs-2591.txt
>
>
> Right now the MiniDFS builder is somewhat inflexible - it just takes a "numNameNodes" parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs -- eg to test the case where there is one nameservice that is HA and another which is not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2591) HA: MiniDFSCluster support to mix and
match federation with HA
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2591:
------------------------------
Attachment: hdfs-2591-v2.txt
v2 patch fixes a number of issues discovered trying to build HA test cases on top of this. In particular, it has code to set up shared edits directories (to be enabled by HDFS-1971), as well as initialize the SBNN by copying one of the formatted namedirs from the active NN. (otherwise the two NNs would have different nsids/bpids and not be able to failover).
> HA: MiniDFSCluster support to mix and match federation with HA
> --------------------------------------------------------------
>
> Key: HDFS-2591
> URL: https://issues.apache.org/jira/browse/HDFS-2591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2591-v2.txt, hdfs-2591.txt
>
>
> Right now the MiniDFS builder is somewhat inflexible - it just takes a "numNameNodes" parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs -- eg to test the case where there is one nameservice that is HA and another which is not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2591) HA: MiniDFSCluster support to mix
and match federation with HA
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159633#comment-13159633 ]
Eli Collins commented on HDFS-2591:
-----------------------------------
+1 the delta lgtm
> HA: MiniDFSCluster support to mix and match federation with HA
> --------------------------------------------------------------
>
> Key: HDFS-2591
> URL: https://issues.apache.org/jira/browse/HDFS-2591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2591-v2.txt, hdfs-2591-v3.txt, hdfs-2591.txt
>
>
> Right now the MiniDFS builder is somewhat inflexible - it just takes a "numNameNodes" parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs -- eg to test the case where there is one nameservice that is HA and another which is not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2591) HA: MiniDFSCluster support to mix
and match federation with HA
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156539#comment-13156539 ]
Todd Lipcon commented on HDFS-2591:
-----------------------------------
(this is still a work in progress - testing it by updating the latest HDFS-1975 patch to run on top of this infrastructure, and identifying a few missing pieces)
> HA: MiniDFSCluster support to mix and match federation with HA
> --------------------------------------------------------------
>
> Key: HDFS-2591
> URL: https://issues.apache.org/jira/browse/HDFS-2591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2591.txt
>
>
> Right now the MiniDFS builder is somewhat inflexible - it just takes a "numNameNodes" parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs -- eg to test the case where there is one nameservice that is HA and another which is not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2591) HA: MiniDFSCluster support to mix and
match federation with HA
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-2591:
------------------------------
Attachment: hdfs-2591.txt
Attached patch introduces a new MiniDFSNNTopology class which is used to describe the layout of NNs in the cluster. Most unit tests are passing, including those that make federated miniclusters. (ran tests with this applied on top of HDFS-2582, though it may also work without that patch)
> HA: MiniDFSCluster support to mix and match federation with HA
> --------------------------------------------------------------
>
> Key: HDFS-2591
> URL: https://issues.apache.org/jira/browse/HDFS-2591
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, test
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-2591.txt
>
>
> Right now the MiniDFS builder is somewhat inflexible - it just takes a "numNameNodes" parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs -- eg to test the case where there is one nameservice that is HA and another which is not.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira