Complete upgrade of HDP stack via Ambari REST API

[shirsha ray chaudhuri <sh...@gmail.com>]

Hi,

Am trying to resend Arka's mail on automating the Ambari + HDP Upgrade
solution. We are trying to see if someone else in the community has tried
this approach to upgrading the stack.

We are trying to completely automate (without any manual intervention) the
upgrade of stack(HDP/any other supported stack) through Ambari rest APIs. We
searched in Internet and didn't find any complete guide on this automation
process, so we tried to come up with a guide for automation for greater good of
the Ambari community.
If similar things are tried and any feedback or experiences could be shared
then it will be of immense help for us.

Motivation:
In our case the automation is a part of bigger orchestration work. This
automation will also help in scenarios where the end user will not have any
Ambari GUI access

References:
1. Ambari Rest API guide
https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/upgrades.md
2. Blog by Alejandro Fernandez
https://community.hortonworks.com/articles/2473/rolling-upgrade-express-upgrade-in-ambari.html
3. Few trial and errors from our end.

Prerequisites/ Assumptions:
1. The stack is supported by Ambari for upgrade i.e. the stack should be
manually upgradable via Ambari.
2. The Ambari details like Ambari ip, port, login credentials, cluster name,
etc. are available from end user.
3. The stack details like repository, version, OS version, upgrade type, etc.
are available from end user.


Steps:
We are listing the Ambari rest APIs only which can be programed in any
language. The user inputs are marked with <> here. Following are steps and the
respective Rest API/Curl commands for upgrade:

1) Registering the Stack:
The first step is registering the new stack with Ambari. We used the following
curl command to do this and the corresponding json file format is:

               POST  http://<Ambari Ip>:<Ambari port>/api/v1/stacks/<New Stack
Name>/versions/<Stack Version>/repository_versions

               {
               "RepositoryVersions":
               {
                              "repository_version":
"<New_Stack_Repository_Version>",
                              "display_name": "<Stack_Display_Name>"
               },
               "operating_systems": [
               {
                              "OperatingSystems":
                              {
                                             "os_type": "<OS_Type_and_Version>"
                              },
                              "repositories":
                              [
                              {
                                             "Repositories":
                                             {
                                                            "repo_id":
"<Stack_Repository_ID>",
                                                            "repo_name":
"<Stack_Repository_Name>",
                                                            "base_url":
"<Stack_Repository_Base_URL>"
                                             }
                              },
                              {
                                             "Repositories":
                                             {
                                                            "repo_id":
"<Stack_Utils_Repository_ID>",
                                                            "repo_name":
"<Stack_Utils_Repository_Name>",
                                                            "base_url":
"<Stack_Utils_Repository_Base_URL>"
                                             }
                              }
                              ]
               }
               ]
               }

We check the response of this command to confirm that the registration is
successfull or not.
If the stack is already registered then this step can return error. We ignore
this particular error.

2) Installing the Stack:
The next step is installing the new stack on each host. We used the following
curl command to do this and the corresponding json file format is:

               POST http://<Ambari Ip>:<Ambari port>/api/v1/clusters/<Cluster
name>/stack_versions

               {
               "ClusterStackVersions":
               {
                              "stack": "<New_Stack_Name>",
                              "version": "<New_Stack_Version>",
                              "repository_version":
"<New_Stack_Repository_Version>"
               }
               }

3) Check the status of Installation:
Since installation is a time-consuming process, so we continuously poll for the
status of the request (the request Id we got from the previous step) for failed
or completed.
We used the following curl command to do this:

               GET http://<Ambari Ip>:<Ambari port>/api/v1/clusters/<Cluster
name>/requests/<Install request ID>

4) Upgrade of stack:
Once the installation is completed successfully, we supply the command to
Ambari for upgrade of stack.
We used the following curl command to do this and the corresponding json file
format is:

               POST http://<Ambari Ip>:<Ambari port>/api/v1/clusters/<Cluster
name>/upgrades

               {
               "Upgrade":
               {
                              "repository_version":
"<New_Stack_Repository_Version>",
                              "upgrade_type" :
"<Upgrade_Type_Rolling_or_Express>"
               }
               }

5) Check status of Upgrade:
We continuously check for the upgrade item status for failed or holding.
During upgrade, there are multiple instances where the upgrade task will go for
HOLDING (which usually means Ambari need user intervention)
Example: During YARN upgrade, AMBARI will ask user if the YARN job queues are
stopped/started.
When an upgrade_item goes to Holding status, we check the context of the
upgrade_item and take necessary action (Complete/Abort the upgrade_item) to
automate the progress of upgrade.
We do this polling till the upgrade task goes on Holding for Finalize/Downgrade
state. We used the following curl command to do this:

               GET http://<Ambari Ip>:<Ambari port>/api/v1/clusters/<Cluster
name>/upgrades?upgrade_groups/upgrade_items/UpgradeItem/status=HOLDING

and check the context of holding upgrade item.

6) Downgrade or Finalize:
Once the upgrade is successful, Ambari upgrade goes to Holding state. In this
step, the end user have a choice to either Downgrade the stack OR Finalize the
upgrade.
We take the user action on this and progress accordingly.

For Downgrade:
Once the user chooses to downgrade, we first abort the existing upgrade task
which is in holding state by issuing following command:

               POST http://<Ambari Ip>:<Ambari port>/api/v1/clusters/<Cluster
name>/upgrades/<Upgrade id>

               {
               "Upgrade":
               {
                              "request_status":"ABORTED",
                              "suspended":true
               }
               }

Then we issue a new downgrade request as follows:

               POST http://<Ambari Ip>:<Ambari port>/api/v1/clusters/<Cluster
name>/upgrades?downgrade=true

               {
               "Upgrade":
               {
                              "from_version": "<Old_Stack_Version>",
                              "repository_version":
"<Old_Stack_Repository_Version>",
                              "direction": "DOWNGRADE",
                              "upgrade_type":
"<Upgrade_Type_Rolling_or_Express>"
               }
               }

For Finalize:
Once the user chooses to finalize the stack, we complete the upgrade_item,
which is in holding state by following command:

               POST http://<Ambari Ip>:<Ambari port>/api/v1/clusters/<Cluster
name>/upgrades/<Upgrade id>/upgrade_groups/<Upgrade_group
id>/upgrade_items/<Upgrade_item id>

               {
               "UpgradeItem":
               {
                              "status":"COMPLETED"
               }
               }



Thanks and Regards

Shirsha / Arka