You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Robert Nettleton <rn...@hortonworks.com> on 2015/07/24 20:19:16 UTC

Review Request 36786: Update Blueprint deployment to wait for the configuration to be fully resolved prior to first install task

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36786/
-----------------------------------------------------------

Review request for Ambari, Mahadev Konar and Sumit Mohanty.


Bugs: AMBARI-12532
    https://issues.apache.org/jira/browse/AMBARI-12532


Repository: ambari


Description
-------

This patch resolved AMBARI-12532. 

When creating larger clusters (50+ nodes), there is an intermittent failure that occurs, in which the configurations on the cluster are not always completely resolved.  This causes the Blueprint service install and host tasks to sometimes fail, since the expected configuration is not yet present across the entire cluster.

This is potentially a timing issue between the Blueprints processor and the Ambari configuration engine. 

This patch implements the following:
1. Adds a new method to the AmbariContext class, that will wait until all the required configuration types enter the TOPOLOGY_RESOLVED phase, which indicates that the configuration is ready across the entire cluster. 
2. Calls this new method from within the Cluster configuration task.  The wait method will block until the desired configuration status is reached. 
3. Adds new logging code in the Blueprints to show the status of the configuration types and their resolution at cluster deployment time.  Also added logging throughout relevant classes in this area, in order to simplify debugging of concurrency problems in the future. 
4. Adds new unit tests, and updates existing tests to verify this change.


Diffs
-----

  ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 1ded566 
  ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 14b13ef 
  ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java 42676aa 
  ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterConfigurationRequest.java eb583fd 
  ambari-server/src/main/java/org/apache/ambari/server/topology/HostRequest.java 5d76f7a 
  ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 7117b05 
  ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java 4ebf9a9 

Diff: https://reviews.apache.org/r/36786/diff/


Testing
-------

1. Ran the ambari-server unit tests ("mvn clean test"), and all tests were passing

"
Java unit test results:

Results :

Tests run: 3135, Failures: 0, Errors: 0, Skipped: 27


Python tests:

Total run:807
Total errors:0
Total failures:0
OK
"

2. Deployed a 3-node HDFS NameNode HA cluster using Blueprints, and verified that this deployment completed successfully, and that the new expected logging output is present in ambari-server.log.


Thanks,

Robert Nettleton


Re: Review Request 36786: Update Blueprint deployment to wait for the configuration to be fully resolved prior to first install task

Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36786/#review92955
-----------------------------------------------------------



ambari-server/src/main/java/org/apache/ambari/server/topology/HostRequest.java (line 90)
<https://reviews.apache.org/r/36786/#comment147221>

    Nice find :-)


- Sumit Mohanty


On July 24, 2015, 6:19 p.m., Robert Nettleton wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36786/
> -----------------------------------------------------------
> 
> (Updated July 24, 2015, 6:19 p.m.)
> 
> 
> Review request for Ambari, Mahadev Konar and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-12532
>     https://issues.apache.org/jira/browse/AMBARI-12532
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> This patch resolved AMBARI-12532. 
> 
> When creating larger clusters (50+ nodes), there is an intermittent failure that occurs, in which the configurations on the cluster are not always completely resolved.  This causes the Blueprint service install and host tasks to sometimes fail, since the expected configuration is not yet present across the entire cluster.
> 
> This is potentially a timing issue between the Blueprints processor and the Ambari configuration engine. 
> 
> This patch implements the following:
> 1. Adds a new method to the AmbariContext class, that will wait until all the required configuration types enter the TOPOLOGY_RESOLVED phase, which indicates that the configuration is ready across the entire cluster. 
> 2. Calls this new method from within the Cluster configuration task.  The wait method will block until the desired configuration status is reached. 
> 3. Adds new logging code in the Blueprints to show the status of the configuration types and their resolution at cluster deployment time.  Also added logging throughout relevant classes in this area, in order to simplify debugging of concurrency problems in the future. 
> 4. Adds new unit tests, and updates existing tests to verify this change.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 1ded566 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 14b13ef 
>   ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java 42676aa 
>   ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterConfigurationRequest.java eb583fd 
>   ambari-server/src/main/java/org/apache/ambari/server/topology/HostRequest.java 5d76f7a 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 7117b05 
>   ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java 4ebf9a9 
> 
> Diff: https://reviews.apache.org/r/36786/diff/
> 
> 
> Testing
> -------
> 
> 1. Ran the ambari-server unit tests ("mvn clean test"), and all tests were passing
> 
> "
> Java unit test results:
> 
> Results :
> 
> Tests run: 3135, Failures: 0, Errors: 0, Skipped: 27
> 
> 
> Python tests:
> 
> Total run:807
> Total errors:0
> Total failures:0
> OK
> "
> 
> 2. Deployed a 3-node HDFS NameNode HA cluster using Blueprints, and verified that this deployment completed successfully, and that the new expected logging output is present in ambari-server.log.
> 
> 
> Thanks,
> 
> Robert Nettleton
> 
>


Re: Review Request 36786: Update Blueprint deployment to wait for the configuration to be fully resolved prior to first install task

Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36786/#review92957
-----------------------------------------------------------

Ship it!


Ship It!

- Sumit Mohanty


On July 24, 2015, 6:19 p.m., Robert Nettleton wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36786/
> -----------------------------------------------------------
> 
> (Updated July 24, 2015, 6:19 p.m.)
> 
> 
> Review request for Ambari, Mahadev Konar and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-12532
>     https://issues.apache.org/jira/browse/AMBARI-12532
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> This patch resolved AMBARI-12532. 
> 
> When creating larger clusters (50+ nodes), there is an intermittent failure that occurs, in which the configurations on the cluster are not always completely resolved.  This causes the Blueprint service install and host tasks to sometimes fail, since the expected configuration is not yet present across the entire cluster.
> 
> This is potentially a timing issue between the Blueprints processor and the Ambari configuration engine. 
> 
> This patch implements the following:
> 1. Adds a new method to the AmbariContext class, that will wait until all the required configuration types enter the TOPOLOGY_RESOLVED phase, which indicates that the configuration is ready across the entire cluster. 
> 2. Calls this new method from within the Cluster configuration task.  The wait method will block until the desired configuration status is reached. 
> 3. Adds new logging code in the Blueprints to show the status of the configuration types and their resolution at cluster deployment time.  Also added logging throughout relevant classes in this area, in order to simplify debugging of concurrency problems in the future. 
> 4. Adds new unit tests, and updates existing tests to verify this change.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 1ded566 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 14b13ef 
>   ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java 42676aa 
>   ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterConfigurationRequest.java eb583fd 
>   ambari-server/src/main/java/org/apache/ambari/server/topology/HostRequest.java 5d76f7a 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 7117b05 
>   ambari-server/src/test/java/org/apache/ambari/server/topology/AmbariContextTest.java 4ebf9a9 
> 
> Diff: https://reviews.apache.org/r/36786/diff/
> 
> 
> Testing
> -------
> 
> 1. Ran the ambari-server unit tests ("mvn clean test"), and all tests were passing
> 
> "
> Java unit test results:
> 
> Results :
> 
> Tests run: 3135, Failures: 0, Errors: 0, Skipped: 27
> 
> 
> Python tests:
> 
> Total run:807
> Total errors:0
> Total failures:0
> OK
> "
> 
> 2. Deployed a 3-node HDFS NameNode HA cluster using Blueprints, and verified that this deployment completed successfully, and that the new expected logging output is present in ambari-server.log.
> 
> 
> Thanks,
> 
> Robert Nettleton
> 
>