You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Robert Nettleton <rn...@hortonworks.com> on 2014/12/20 00:11:31 UTC
Review Request 29276: Implements support for deploying HDFS NameNode
High-Availability (HA) clusters using a Blueprint
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29276/
-----------------------------------------------------------
Review request for Ambari, Jaimin Jetly, John Speidel, Robert Levas, and Yusaku Sako.
Bugs: AMBARI-8771
https://issues.apache.org/jira/browse/AMBARI-8771
Repository: ambari
Description
-------
This patch resolves AMBARI-8771.
In previous releases of Ambari Blueprints, it was not possible to deploy an HDFS NameNode HA cluster using a Blueprint. The existing HA support in Ambari in the UI supported adding HA support to a running cluster, but Ambari did not support
creating HA clusters from the initial deployment of the cluster.
This patch adds support for deploying HDFS NameNode HA clusters via the following changes:
- Modifies the HDFS stack service scripts to handle the startup and
configuration requirements of an initial HDFS HA install, including
handling the startup of the active and standby nodes
- Implements support for two new configuration properties in hadoop-env
that allow the user to specify the initial states of the namenodes, by
declaring the host name of the active and standby nodes. These
properties are used by the stack scripts in order to run the setup
specific to each node type. If these properties are not set by
the user, the Blueprints processor will choose defaults for the
active and stanbdy nodes in the HDFS cluster.
- Updates the Blueprint configuration code to handle quorum journal URL
properties, which use a slightly different syntax than other Hadoop
configuration properties.
- Updates the Blueprint validation code to support a NameNode HA deployment
that does not include the SECONDARY_NAMENODE. Currently, our stacks
require the secondary namenode, and there is no way to express a
cardinality that is optional based on HA status, so this handling
now occurs in the Blueprint processor.
- Adds/updates unit tests for all Java changes in this patch
- Adds/updates unit tests for all Python stack changes in this patch
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BaseBlueprintProcessor.java 9cfb635
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java c4071d4
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py e8dbc59
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py c9c81bb
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py ee8b418
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BaseBlueprintProcessorTest.java 8540d8b
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java abc0b15
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 9c65210
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_zkfc.py c358e9e
ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json PRE-CREATION
ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json PRE-CREATION
Diff: https://reviews.apache.org/r/29276/diff/
Testing
-------
1. Ran the ambari-server unit tests: "Tests run: 2451, Failures: 0, Errors: 0, Skipped: 13"
2. Manually verified that I can start a 3-node HDFS HA Cluster with a Blueprint. I used the Ambari UI to verify that the cluster started via the Blueprint sets up the active and standby nodes. I've also used the "hdfs haadmin" tool to verify the status of the NameNodes setup during a Blueprint-based HA install.
3. Several times this week, I also started up a cluster with my changes applied, and then used the UI Wizard to setup NameNode HA, to make sure the stack changes don't break the UI support for adding HA to an existing cluster.
Thanks,
Robert Nettleton
Re: Review Request 29276: Implements support for deploying HDFS
NameNode High-Availability (HA) clusters using a Blueprint
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29276/#review65751
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Levas
On Dec. 19, 2014, 6:11 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29276/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2014, 6:11 p.m.)
>
>
> Review request for Ambari, Jaimin Jetly, John Speidel, Robert Levas, and Yusaku Sako.
>
>
> Bugs: AMBARI-8771
> https://issues.apache.org/jira/browse/AMBARI-8771
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch resolves AMBARI-8771.
>
> In previous releases of Ambari Blueprints, it was not possible to deploy an HDFS NameNode HA cluster using a Blueprint. The existing HA support in Ambari in the UI supported adding HA support to a running cluster, but Ambari did not support
> creating HA clusters from the initial deployment of the cluster.
>
> This patch adds support for deploying HDFS NameNode HA clusters via the following changes:
>
> - Modifies the HDFS stack service scripts to handle the startup and
> configuration requirements of an initial HDFS HA install, including
> handling the startup of the active and standby nodes
> - Implements support for two new configuration properties in hadoop-env
> that allow the user to specify the initial states of the namenodes, by
> declaring the host name of the active and standby nodes. These
> properties are used by the stack scripts in order to run the setup
> specific to each node type. If these properties are not set by
> the user, the Blueprints processor will choose defaults for the
> active and stanbdy nodes in the HDFS cluster.
> - Updates the Blueprint configuration code to handle quorum journal URL
> properties, which use a slightly different syntax than other Hadoop
> configuration properties.
> - Updates the Blueprint validation code to support a NameNode HA deployment
> that does not include the SECONDARY_NAMENODE. Currently, our stacks
> require the secondary namenode, and there is no way to express a
> cardinality that is optional based on HA status, so this handling
> now occurs in the Blueprint processor.
> - Adds/updates unit tests for all Java changes in this patch
> - Adds/updates unit tests for all Python stack changes in this patch
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BaseBlueprintProcessor.java 9cfb635
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java c4071d4
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py e8dbc59
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py c9c81bb
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py ee8b418
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BaseBlueprintProcessorTest.java 8540d8b
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java abc0b15
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 9c65210
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_zkfc.py c358e9e
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json PRE-CREATION
>
> Diff: https://reviews.apache.org/r/29276/diff/
>
>
> Testing
> -------
>
> 1. Ran the ambari-server unit tests: "Tests run: 2451, Failures: 0, Errors: 0, Skipped: 13"
> 2. Manually verified that I can start a 3-node HDFS HA Cluster with a Blueprint. I used the Ambari UI to verify that the cluster started via the Blueprint sets up the active and standby nodes. I've also used the "hdfs haadmin" tool to verify the status of the NameNodes setup during a Blueprint-based HA install.
> 3. Several times this week, I also started up a cluster with my changes applied, and then used the UI Wizard to setup NameNode HA, to make sure the stack changes don't break the UI support for adding HA to an existing cluster.
>
>
> Thanks,
>
> Robert Nettleton
>
>
Re: Review Request 29276: Implements support for deploying HDFS
NameNode High-Availability (HA) clusters using a Blueprint
Posted by John Speidel <js...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29276/#review65744
-----------------------------------------------------------
Ship it!
Ship It!
- John Speidel
On Dec. 19, 2014, 11:11 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29276/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2014, 11:11 p.m.)
>
>
> Review request for Ambari, Jaimin Jetly, John Speidel, Robert Levas, and Yusaku Sako.
>
>
> Bugs: AMBARI-8771
> https://issues.apache.org/jira/browse/AMBARI-8771
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch resolves AMBARI-8771.
>
> In previous releases of Ambari Blueprints, it was not possible to deploy an HDFS NameNode HA cluster using a Blueprint. The existing HA support in Ambari in the UI supported adding HA support to a running cluster, but Ambari did not support
> creating HA clusters from the initial deployment of the cluster.
>
> This patch adds support for deploying HDFS NameNode HA clusters via the following changes:
>
> - Modifies the HDFS stack service scripts to handle the startup and
> configuration requirements of an initial HDFS HA install, including
> handling the startup of the active and standby nodes
> - Implements support for two new configuration properties in hadoop-env
> that allow the user to specify the initial states of the namenodes, by
> declaring the host name of the active and standby nodes. These
> properties are used by the stack scripts in order to run the setup
> specific to each node type. If these properties are not set by
> the user, the Blueprints processor will choose defaults for the
> active and stanbdy nodes in the HDFS cluster.
> - Updates the Blueprint configuration code to handle quorum journal URL
> properties, which use a slightly different syntax than other Hadoop
> configuration properties.
> - Updates the Blueprint validation code to support a NameNode HA deployment
> that does not include the SECONDARY_NAMENODE. Currently, our stacks
> require the secondary namenode, and there is no way to express a
> cardinality that is optional based on HA status, so this handling
> now occurs in the Blueprint processor.
> - Adds/updates unit tests for all Java changes in this patch
> - Adds/updates unit tests for all Python stack changes in this patch
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BaseBlueprintProcessor.java 9cfb635
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java c4071d4
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py e8dbc59
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py c9c81bb
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py ee8b418
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BaseBlueprintProcessorTest.java 8540d8b
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java abc0b15
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 9c65210
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_zkfc.py c358e9e
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json PRE-CREATION
>
> Diff: https://reviews.apache.org/r/29276/diff/
>
>
> Testing
> -------
>
> 1. Ran the ambari-server unit tests: "Tests run: 2451, Failures: 0, Errors: 0, Skipped: 13"
> 2. Manually verified that I can start a 3-node HDFS HA Cluster with a Blueprint. I used the Ambari UI to verify that the cluster started via the Blueprint sets up the active and standby nodes. I've also used the "hdfs haadmin" tool to verify the status of the NameNodes setup during a Blueprint-based HA install.
> 3. Several times this week, I also started up a cluster with my changes applied, and then used the UI Wizard to setup NameNode HA, to make sure the stack changes don't break the UI support for adding HA to an existing cluster.
>
>
> Thanks,
>
> Robert Nettleton
>
>
Re: Review Request 29276: Implements support for deploying HDFS
NameNode High-Availability (HA) clusters using a Blueprint
Posted by Robert Nettleton <rn...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29276/#review65794
-----------------------------------------------------------
Merged to trunk
Test results:
Results :
Tests run: 2451, Failures: 0, Errors: 0, Skipped: 13
- Robert Nettleton
On Dec. 19, 2014, 11:11 p.m., Robert Nettleton wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29276/
> -----------------------------------------------------------
>
> (Updated Dec. 19, 2014, 11:11 p.m.)
>
>
> Review request for Ambari, Jaimin Jetly, John Speidel, Robert Levas, and Yusaku Sako.
>
>
> Bugs: AMBARI-8771
> https://issues.apache.org/jira/browse/AMBARI-8771
>
>
> Repository: ambari
>
>
> Description
> -------
>
> This patch resolves AMBARI-8771.
>
> In previous releases of Ambari Blueprints, it was not possible to deploy an HDFS NameNode HA cluster using a Blueprint. The existing HA support in Ambari in the UI supported adding HA support to a running cluster, but Ambari did not support
> creating HA clusters from the initial deployment of the cluster.
>
> This patch adds support for deploying HDFS NameNode HA clusters via the following changes:
>
> - Modifies the HDFS stack service scripts to handle the startup and
> configuration requirements of an initial HDFS HA install, including
> handling the startup of the active and standby nodes
> - Implements support for two new configuration properties in hadoop-env
> that allow the user to specify the initial states of the namenodes, by
> declaring the host name of the active and standby nodes. These
> properties are used by the stack scripts in order to run the setup
> specific to each node type. If these properties are not set by
> the user, the Blueprints processor will choose defaults for the
> active and stanbdy nodes in the HDFS cluster.
> - Updates the Blueprint configuration code to handle quorum journal URL
> properties, which use a slightly different syntax than other Hadoop
> configuration properties.
> - Updates the Blueprint validation code to support a NameNode HA deployment
> that does not include the SECONDARY_NAMENODE. Currently, our stacks
> require the secondary namenode, and there is no way to express a
> cardinality that is optional based on HA status, so this handling
> now occurs in the Blueprint processor.
> - Adds/updates unit tests for all Java changes in this patch
> - Adds/updates unit tests for all Python stack changes in this patch
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BaseBlueprintProcessor.java 9cfb635
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java c4071d4
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py e8dbc59
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params.py c9c81bb
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py ee8b418
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BaseBlueprintProcessorTest.java 8540d8b
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java abc0b15
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py 9c65210
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_zkfc.py c358e9e
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_active_node.json PRE-CREATION
> ambari-server/src/test/python/stacks/2.0.6/configs/ha_bootstrap_standby_node.json PRE-CREATION
>
> Diff: https://reviews.apache.org/r/29276/diff/
>
>
> Testing
> -------
>
> 1. Ran the ambari-server unit tests: "Tests run: 2451, Failures: 0, Errors: 0, Skipped: 13"
> 2. Manually verified that I can start a 3-node HDFS HA Cluster with a Blueprint. I used the Ambari UI to verify that the cluster started via the Blueprint sets up the active and standby nodes. I've also used the "hdfs haadmin" tool to verify the status of the NameNodes setup during a Blueprint-based HA install.
> 3. Several times this week, I also started up a cluster with my changes applied, and then used the UI Wizard to setup NameNode HA, to make sure the stack changes don't break the UI support for adding HA to an existing cluster.
>
>
> Thanks,
>
> Robert Nettleton
>
>