Extending an existing cluster via API calls

[jo...@post.ch]

I'm looking to extend an existing cluster via API calls. The cluster was configured to have hostgroups 2 (Ambari server with all necessary infrastructure) and hostgroup 1 (Zookeeper etc).

According to the documentation, it looks like I should be able to post the following JSON to /api/v1/clusters/idp/hosts/<new_node_name>

{"blueprint": "cluster1", "host_group": "host_group_1"}

I would expect Ambari to pick up the services as defined in the blueprint for host_group_1, and start installing them automatically (as, again, described in the blueprint).

This isn't what happens. Restarting ambari-agent on the node immediately after calling the above API yields the log below:



18 May 2018 14:14:44,473  INFO [ambari-client-thread-73] TopologyManager:485 - TopologyManager.scaleHosts: Entering
18 May 2018 14:14:44,474  INFO [ambari-client-thread-73] ClusterTopologyImpl:158 - ClusterTopologyImpl.addHostTopology: added host = node5 to host group = host_group_1
18 May 2018 14:14:44,480  INFO [ambari-client-thread-73] HostRequest:205 - Skipping Start task creation since provision action = INSTALL_ONLY
18 May 2018 14:14:44,483  INFO [ambari-client-thread-73] HostRequest:244 - Skipping create of START task for ZOOKEEPER_SERVER on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1.
18 May 2018 14:14:44,486  INFO [ambari-client-thread-73] HostRequest:244 - Skipping create of START task for ZOOKEEPER_CLIENT on PENDING HOST ASSIGNMENT : HOSTGROUP=host_group_1.
18 May 2018 14:14:44,486  INFO [ambari-client-thread-73] HostRequest:99 - HostRequest: Created request for host: node5
18 May 2018 14:14:44,486  INFO [ambari-client-thread-73] LogicalRequest:437 - LogicalRequest.createHostRequests: all host requests size 1 , outstanding requests size = 0
18 May 2018 14:14:44,491  INFO [ambari-client-thread-73] TopologyManager:923 - TopologyManager.createLogicalRequest: created LogicalRequest with ID = 51 and completed persistence of this request.
18 May 2018 14:14:44,495  INFO [ambari-client-thread-73] TopologyManager:845 - TopologyManager.processRequest: Entering
18 May 2018 14:14:44,496  INFO [ambari-client-thread-73] TopologyManager:863 - TopologyManager.processRequest: host name = node5 is mapped to LogicalRequest ID = 51 and will be removed from the reserved hosts.
18 May 2018 14:14:44,496  INFO [ambari-client-thread-73] TopologyManager:876 - TopologyManager.processRequest: offering host name = node5 to LogicalRequest ID = 51
18 May 2018 14:14:44,496  INFO [ambari-client-thread-73] LogicalRequest:101 - LogicalRequest.offer: attempting to match a request to a request for a reserved host to hostname = node5
18 May 2018 14:14:44,497  INFO [ambari-client-thread-73] LogicalRequest:110 - LogicalRequest.offer: request mapping ACCEPTED for host = node5
18 May 2018 14:14:44,497  INFO [ambari-client-thread-73] LogicalRequest:113 - LogicalRequest.offer returning response, reservedHost list size = 0
18 May 2018 14:14:44,497  INFO [ambari-client-thread-73] TopologyManager:886 - TopologyManager.processRequest: host name = node5 was ACCEPTED by LogicalRequest ID = 51 , host has been removed from available hosts.
18 May 2018 14:14:44,498  INFO [ambari-client-thread-73] ClusterTopologyImpl:158 - ClusterTopologyImpl.addHostTopology: added host = node5 to host group = host_group_1
18 May 2018 14:14:44,502  INFO [ambari-client-thread-73] TopologyManager:963 - TopologyManager.processAcceptedHostOffer: queue tasks for host = node5 which responded ACCEPTED
18 May 2018 14:14:44,502  INFO [ambari-client-thread-73] TopologyManager:988 - TopologyManager.processAcceptedHostOffer: queueing tasks for host = node5
18 May 2018 14:14:44,503  INFO [ambari-client-thread-73] TopologyManager:977 - Creating TopologyTaskExecutorService for clusterId: 2
18 May 2018 14:14:44,503  INFO [ambari-client-thread-73] TopologyManager:904 - TopologyManager.processRequest: not all required hosts have been matched, so adding LogicalRequest ID = 51 to outstanding requests
18 May 2018 14:14:44,504  INFO [ambari-client-thread-73] AbstractProviderModule:864 - Resetting property provider maps to reflect changes in cluster state
18 May 2018 14:14:46,873  INFO [ambari-client-thread-74] MetricsCollectorHAManager:54 - Adding collector host : node1 to cluster : cluster
18 May 2018 14:15:05,924  WARN [qtp-ambari-agent-253] SecurityFilter:103 - Request https://node1:8440/ca doesn't match any pattern.
18 May 2018 14:15:05,925  WARN [qtp-ambari-agent-253] SecurityFilter:62 - This request is not allowed on this port: https://node1:8440/ca
18 May 2018 14:15:08,122  INFO [qtp-ambari-agent-255] HeartBeatHandler:385 - agentOsType = redhat7
18 May 2018 14:15:08,128  INFO [qtp-ambari-agent-255] HostImpl:334 - Received host registration, host=[hostname=node5,fqdn=node5,domain=,architecture=x86_64,processorcount=8,physicalprocessorcount=8,osname=redhat,osversion=7.5,osfamily=redhat,memory=32764508,uptime_hours=360,mounts=(available=8617972,mountpoint=/,used=1849356,percent=18%,size=10467328,device=/dev/mapper/docker-253:9-33585216-27511a71bf68d8c446b67e10d9b5600e40d1d73efca041857be3553d279ac8ec,type=xfs)]
, registrationTime=1526645708122, agentVersion=2.6.1.0
18 May 2018 14:15:08,128  INFO [qtp-ambari-agent-255] TopologyManager:637 - TopologyManager.onHostRegistered: Entering
18 May 2018 14:15:08,128  INFO [qtp-ambari-agent-255] TopologyManager:639 - TopologyManager.onHostRegistered: host = node5 is already associated with the cluster or is currently being processed
18 May 2018 14:15:08,130  INFO [qtp-ambari-agent-255] HeartBeatHandler:464 - Recovery configuration set to RecoveryConfig{, type=AUTO_START, maxCount=6, windowInMinutes=60, retryGap=5, maxLifetimeCount=1024, components=, recoveryTimestamp=1526645708130}



I'm probably missing something here. Could someone point me into the right direction as to service inheritance by hostgroup (or, related documentation) ?

Regards





Johannes Meixner

Post CH AG
Informationstechnologie
DWH & Big Data Solutions
Webergutstrasse 5
3030 Bern (Zollikofen)

Internet: http://www.post.ch
Tel.: +41 58 341 40 77