You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by John Speidel <js...@hortonworks.com> on 2015/04/26 04:27:25 UTC
Review Request 33559: Initial merge of Advanced API Cluster
Provisioning Functionality
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33559/
-----------------------------------------------------------
Review request for Ambari and Mahadev Konar.
Bugs: AMBARI-10750
https://issues.apache.org/jira/browse/AMBARI-10750
Repository: ambari
Description
-------
API based (blueprint) cluster provisioning modifications and enhancements which will provide for more flexible, scalable and robust cluster provisioning.
This will include API and backend changes.
- Allow a host count to be specified in the cluster creation template instead of host names. This is documented in https://issues.apache.org/jira/browse/AMBARI-6275
- Allow cluster creation or scaling to be started via the REST API prior to all/any hosts being available. As hosts register with Ambari server they will be matched to request host groups and provisioned according to the requested topology
- Allow host predicates to be specified along with host count to provide more flexibility in matching hosts to host groups. This will allow for host flavors where different host groups are matched to different host flavors
-Break up the current monolithic provisioning request into a request for each host operation. For example, install on host A, start on host A, install on hostB, etc. This will allow hosts to make progress even when another host encounters a failure.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionManager.java 097a0f1
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 662a545
ambari-server/src/main/java/org/apache/ambari/server/api/query/render/ClusterBlueprintRenderer.java a5cdfe0
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java 11a51c4
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java d54a14d
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 0743629
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java 8b767d7
ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java 184c8db
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java f198523
ambari-server/src/main/java/org/apache/ambari/server/controller/ShortTaskStatus.java 3155746
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractResourceProvider.java 652cae3
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BaseBlueprintProcessor.java 73ea1a5
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 9c3266a
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintResourceProvider.java 03acd40
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Cardinality.java 74e594f
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java 2c43e1a
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterResourceProvider.java 0b5bb8e
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ComponentResourceProvider.java dea90f6
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ConfigurationTopologyException.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExportBlueprintRequest.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostComponentResourceProvider.java 4c37d5b
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostGroup.java 303bd15
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 8c51177
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequest.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java 1d5d90a
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ScaleClusterRequest.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceResourceProvider.java e715d42
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Stack.java 9ef13ba
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java fd6b751
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/BlueprintDAO.java 9b58422
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/BlueprintEntity.java 71a64af
ambari-server/src/main/java/org/apache/ambari/server/stack/NoSuchStackException.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 19fe2dd
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java 39219a3
ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java 50d762e
ambari-server/src/main/java/org/apache/ambari/server/topology/Blueprint.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintFactory.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintImpl.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidator.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/Cardinality.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterConfigurationRequest.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopology.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/Configuration.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/ConfigurationFactory.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroup.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroupImpl.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroupInfo.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/HostOfferResponse.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/HostRequest.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/InvalidTopologyException.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/InvalidTopologyTemplateException.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/LogicalRequest.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/NoSuchBlueprintException.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/NoSuchHostGroupException.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/RequiredPasswordValidator.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyRequest.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyRequestFactory.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyRequestFactoryImpl.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyTask.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyValidator.java PRE-CREATION
ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 9bf2ac4
ambari-server/src/test/java/org/apache/ambari/server/api/query/render/ClusterBlueprintRendererTest.java 15280b9
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java 33bb830
ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java cba560a
ambari-server/src/test/java/org/apache/ambari/server/controller/BackgroundCustomCommandExecutionTest.java 4446dfd
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java cf903d0
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BaseBlueprintProcessorTest.java 2494219
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 6c26b06
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintResourceProviderTest.java 1fa4a4d
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProviderTest.java e138196
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterResourceProviderTest.java 0d888e3
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostComponentResourceProviderTest.java 8d1d20d
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequestTest.java PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java 64210db
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java d79f809
ambari-server/src/test/java/org/apache/ambari/server/orm/entities/BlueprintEntityTest.java 2587f6d
ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintFactoryTest.java PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintImplTest.java PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/topology/ClusterTopologyImplTest.java PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/topology/ConfigurationFactoryTest.java PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/topology/RequiredPasswordValidatorTest.java PRE-CREATION
ambari-server/src/test/java/org/apache/ambari/server/utils/TestStageUtils.java 59cbd2b
Diff: https://reviews.apache.org/r/33559/diff/
Testing
-------
Functional Testing:
- Provison and scale a cluster via api using explicit host name, host count and host predicates. Some hosts were registed prior to the requests and some after. Monitored the request via the request api which was returned in the async response.
- Deployed a cluster via the UI.
Unit Testing:
- All unit tests pass
- New unit tests added
Results :
Tests run: 2899, Failures: 0, Errors: 0, Skipped: 17
...
OK
----------------------------------------------------------------------
Total run:685
Total errors:0
Total failures:0
Thanks,
John Speidel
Re: Review Request 33559: Initial merge of Advanced API Cluster
Provisioning Functionality
Posted by Mahadev Konar <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33559/#review81661
-----------------------------------------------------------
Ship it!
Ship It!
- Mahadev Konar
On April 26, 2015, 2:27 a.m., John Speidel wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33559/
> -----------------------------------------------------------
>
> (Updated April 26, 2015, 2:27 a.m.)
>
>
> Review request for Ambari and Mahadev Konar.
>
>
> Bugs: AMBARI-10750
> https://issues.apache.org/jira/browse/AMBARI-10750
>
>
> Repository: ambari
>
>
> Description
> -------
>
> API based (blueprint) cluster provisioning modifications and enhancements which will provide for more flexible, scalable and robust cluster provisioning.
>
> This will include API and backend changes.
> - Allow a host count to be specified in the cluster creation template instead of host names. This is documented in https://issues.apache.org/jira/browse/AMBARI-6275
> - Allow cluster creation or scaling to be started via the REST API prior to all/any hosts being available. As hosts register with Ambari server they will be matched to request host groups and provisioned according to the requested topology
> - Allow host predicates to be specified along with host count to provide more flexibility in matching hosts to host groups. This will allow for host flavors where different host groups are matched to different host flavors
> -Break up the current monolithic provisioning request into a request for each host operation. For example, install on host A, start on host A, install on hostB, etc. This will allow hosts to make progress even when another host encounters a failure.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ActionManager.java 097a0f1
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/HostRoleCommand.java 662a545
> ambari-server/src/main/java/org/apache/ambari/server/api/query/render/ClusterBlueprintRenderer.java a5cdfe0
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java 11a51c4
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java d54a14d
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 0743629
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java 8b767d7
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java 184c8db
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelper.java f198523
> ambari-server/src/main/java/org/apache/ambari/server/controller/ShortTaskStatus.java 3155746
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractResourceProvider.java 652cae3
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BaseBlueprintProcessor.java 73ea1a5
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java 9c3266a
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintResourceProvider.java 03acd40
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Cardinality.java 74e594f
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java 2c43e1a
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClusterResourceProvider.java 0b5bb8e
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ComponentResourceProvider.java dea90f6
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ConfigurationTopologyException.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ExportBlueprintRequest.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostComponentResourceProvider.java 4c37d5b
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostGroup.java 303bd15
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/HostResourceProvider.java 8c51177
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequest.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/RequestResourceProvider.java 1d5d90a
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ScaleClusterRequest.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ServiceResourceProvider.java e715d42
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/Stack.java 9ef13ba
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StageResourceProvider.java fd6b751
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/BlueprintDAO.java 9b58422
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/BlueprintEntity.java 71a64af
> ambari-server/src/main/java/org/apache/ambari/server/stack/NoSuchStackException.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/state/Cluster.java 19fe2dd
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClusterImpl.java 39219a3
> ambari-server/src/main/java/org/apache/ambari/server/state/host/HostImpl.java 50d762e
> ambari-server/src/main/java/org/apache/ambari/server/topology/Blueprint.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintFactory.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintImpl.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidator.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/BlueprintValidatorImpl.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/Cardinality.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterConfigurationRequest.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopology.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/ClusterTopologyImpl.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/Configuration.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/ConfigurationFactory.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroup.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroupImpl.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/HostGroupInfo.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/HostOfferResponse.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/HostRequest.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/InvalidTopologyException.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/InvalidTopologyTemplateException.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/LogicalRequest.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/NoSuchBlueprintException.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/NoSuchHostGroupException.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/RequiredPasswordValidator.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyManager.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyRequest.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyRequestFactory.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyRequestFactoryImpl.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyTask.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/topology/TopologyValidator.java PRE-CREATION
> ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 9bf2ac4
> ambari-server/src/test/java/org/apache/ambari/server/api/query/render/ClusterBlueprintRendererTest.java 15280b9
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelperTest.java 33bb830
> ambari-server/src/test/java/org/apache/ambari/server/controller/AmbariManagementControllerTest.java cba560a
> ambari-server/src/test/java/org/apache/ambari/server/controller/BackgroundCustomCommandExecutionTest.java 4446dfd
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java cf903d0
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BaseBlueprintProcessorTest.java 2494219
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessorTest.java 6c26b06
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/BlueprintResourceProviderTest.java 1fa4a4d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProviderTest.java e138196
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterResourceProviderTest.java 0d888e3
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/HostComponentResourceProviderTest.java 8d1d20d
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ProvisionClusterRequestTest.java PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/RequestResourceProviderTest.java 64210db
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java d79f809
> ambari-server/src/test/java/org/apache/ambari/server/orm/entities/BlueprintEntityTest.java 2587f6d
> ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintFactoryTest.java PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/topology/BlueprintImplTest.java PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/topology/ClusterTopologyImplTest.java PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/topology/ConfigurationFactoryTest.java PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/topology/RequiredPasswordValidatorTest.java PRE-CREATION
> ambari-server/src/test/java/org/apache/ambari/server/utils/TestStageUtils.java 59cbd2b
>
> Diff: https://reviews.apache.org/r/33559/diff/
>
>
> Testing
> -------
>
> Functional Testing:
> - Provison and scale a cluster via api using explicit host name, host count and host predicates. Some hosts were registed prior to the requests and some after. Monitored the request via the request api which was returned in the async response.
> - Deployed a cluster via the UI.
>
> Unit Testing:
> - All unit tests pass
> - New unit tests added
>
> Results :
>
> Tests run: 2899, Failures: 0, Errors: 0, Skipped: 17
> ...
> OK
> ----------------------------------------------------------------------
> Total run:685
> Total errors:0
> Total failures:0
>
>
> Thanks,
>
> John Speidel
>
>