You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by mm...@apache.org on 2019/11/13 18:36:48 UTC
[metron] branch master updated: METRON-2239 Metron Automated backup
and restore (mmiklavc) closes apache/metron#1546
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new fca9ebf METRON-2239 Metron Automated backup and restore (mmiklavc) closes apache/metron#1546
fca9ebf is described below
commit fca9ebf0dc945bea485be334e4fcfcbe7ed179cf
Author: mmiklavc <mi...@gmail.com>
AuthorDate: Wed Nov 13 11:36:07 2019 -0700
METRON-2239 Metron Automated backup and restore (mmiklavc) closes apache/metron#1546
---
Upgrade_steps.md | 40 +++++++
.../packaging/docker/rpm-docker/SPECS/metron.spec | 1 +
metron-platform/metron-common/README.md | 30 ++++++
.../src/main/scripts/upgrade_helper.sh | 120 +++++++++++++++++++++
4 files changed, 191 insertions(+)
diff --git a/Upgrade_steps.md b/Upgrade_steps.md
new file mode 100644
index 0000000..7d197b8
--- /dev/null
+++ b/Upgrade_steps.md
@@ -0,0 +1,40 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Upgrade Steps
+General guidance for upgrading Metron
+
+1. Cut-off all inputs to Metron
+1. Monitor processing until all in-flight data is indexed and stored
+1. Stop Metron service
+1. Run the [Metron Upgrade Helper](./metron-platform/metron-common#metron-upgrade-helper) script in backup mode - does these 2 tasks
+ 1. Export zookeeper based metron configs
+ 1. Export ambari-based metron configs
+1. Delete the metron service via Ambari
+1. Upgrade OS, if applicable
+1. Major Hadoop platform upgrade steps (example. using HDP 2.6 to 3.x) - https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade-major/content/ambari_upgrade_guide.html
+ 1. Update Ambari to latest version
+ 1. Update to HDP 3.1 using existing Ambari/HDP update documentation (including Solr/ES etc).
+1. Reinstall Metron mpack using "--force" option
+1. Reinstall Metron service
+1. Turn off Metron service
+1. Tweak any configs required to support new Metron version
+ 1. Minimally, you should update `metron.home` in `Ambari -> Metron -> Configs -> Advanced metron-env -> Metron home` to point to the new Metron home dir.
+1. Run upgrade helper script in restore mode - does these 2 tasks
+ 1. Re-import zookeeper based metron configs
+ 1. Re-import ambari-based zookeeper configs
+1. Turn on Metron service
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index 673ace5..e4b99ca 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -143,6 +143,7 @@ This package installs the Metron common files %{metron_home}
%{metron_home}/bin/stellar
%{metron_home}/bin/cluster_info.py
%{metron_home}/bin/tgt_renew.py
+%{metron_home}/bin/upgrade_helper.sh
%{metron_home}/config/zookeeper/global.json
%attr(0644,root,root) %{metron_home}/lib/metron-common-%{full_version}-uber.jar
diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md
index f3082a5..2cee5dd 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -25,6 +25,7 @@ limitations under the License.
* [Topology Errors](topology-errors)
* [Performance Logging](#performance-logging)
* [Metron Debugging](#metron-debugging)
+* [Metron Upgrade Helper](#metron-upgrade-helper)
# Stellar Language
@@ -471,3 +472,32 @@ Options:
-p DIRECTORY, --hdp_home=DIRECTORY
HDP home directory
```
+
+# Metron Upgrade Helper
+
+A bash script is provided to assist in performing backup and restore operations for Metron Ambari configurations and configurations stored in Zookeeper.
+
+If your Ambari Server is installed on a separate host from Metron, you may need to scp the upgrade_helper.sh script to the Ambari host along with the file `/etc/default/metron`.
+There is an optional argument, `directory_base`, that allows you to specify where you would like backups to be written to and restored from. Be aware that while it's optional, the
+default is to write the data to the directory from which you're executing the script, i.e. `./metron-backup`.
+
+```
+# $METRON_HOME/bin/upgrade_helper.sh -h
+5 args required
+Usage:
+ mode: [backup|restore] - backup will save configs to a directory named "metron-backup". Restore will take those same configs and restore them to Ambari.
+ ambari_address: host and port for Ambari server, e.g. "node1:8080"
+ username: Ambari admin username
+ password: Ambari admin user password
+ cluster_name: hadoop cluster name. Can be found in Ambari under "Admin > Manage Ambari"
+ directory_base: (Optional) root directory location where the backup will be written to and read from. Default is the executing directory, ".", with backup data stored to a subdirectory named "metron-backup"
+```
+
+```
+Examples:
+# backup
+$METRON_HOME/bin/upgrade_helper.sh backup node1:8080 admin admin metron_cluster
+# restore
+$METRON_HOME/bin/upgrade_helper.sh restore node1:8080 admin admin metron_cluster
+```
+
diff --git a/metron-platform/metron-common/src/main/scripts/upgrade_helper.sh b/metron-platform/metron-common/src/main/scripts/upgrade_helper.sh
new file mode 100755
index 0000000..e272756
--- /dev/null
+++ b/metron-platform/metron-common/src/main/scripts/upgrade_helper.sh
@@ -0,0 +1,120 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+if [ "$#" -ne 5 ] || [ "$1" == "-h" ] || [ "$1" == "--help" ]; then
+ echo 5 args required
+ echo Usage:
+ echo " mode: [backup|restore] - backup will save configs to a directory named \"metron-backup\". Restore will take those same configs and restore them to Ambari."
+ echo " ambari_address: host and port for Ambari server, e.g. \"node1:8080\""
+ echo " username: Ambari admin username"
+ echo " password: Ambari admin user password"
+ echo " cluster_name: hadoop cluster name. Can be found in Ambari under \"Admin > Manage Ambari\""
+ echo " directory_base: (Optional) root directory location where the backup will be written to and read from. Default is the executing directory, \".\", with backup data stored to a subdirectory named \"metron-backup\""
+ exit 1
+fi
+
+mode=$1
+ambari_address=$2
+username=$3
+password=$4
+cluster_name=$5
+# optional base directory
+outdir_base=${6:-.}
+
+if [ -f "/etc/default/metron" ]; then
+ source /etc/default/metron
+fi
+
+OUT_DIR=$outdir_base/metron-backup
+AMBARI_CONFIG_DIR=$OUT_DIR/ambari-configs
+ZK_CONFIG_DIR=$OUT_DIR/zk-configs
+
+if [ "$mode" == "backup" ]; then
+ if [ ! -d "$OUT_DIR" ]; then
+ mkdir $OUT_DIR
+ fi
+ if [ ! -d "$AMBARI_CONFIG_DIR" ]; then
+ mkdir $AMBARI_CONFIG_DIR
+ fi
+ if [ ! -d "$ZK_CONFIG_DIR" ]; then
+ mkdir $ZK_CONFIG_DIR
+ fi
+ if [ -f "/var/lib/ambari-server/resources/scripts/configs.py" ]; then
+ echo Backing up Ambari config...
+ for config_type in $(curl -u $username:$password -H "X-Requested-By: ambari" -X GET http://$ambari_address/api/v1/clusters/$cluster_name?fields=Clusters/desired_configs | grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2 | grep metron);
+ do
+ echo Saving $config_type
+ /var/lib/ambari-server/resources/scripts/configs.py -u $username -p $password -a get -l ${ambari_address%:*} -n $cluster_name -c $config_type -f $AMBARI_CONFIG_DIR/${config_type}.json
+ done
+ echo Done backing up Ambari config...
+ else
+ echo Skipping Ambari config backup - Ambari not found on this host
+ fi
+
+ if [ -f "$METRON_HOME/bin/zk_load_configs.sh" ]; then
+ echo Backing up Metron config
+ $METRON_HOME/bin/zk_load_configs.sh -m PULL -o $ZK_CONFIG_DIR -z $ZOOKEEPER -f
+ echo Done backing up Metron config
+ else
+ echo Skipping Metron config backup - Metron not found on this host
+ fi
+elif [ "$mode" == "restore" ]; then
+ if [ ! -d "$OUT_DIR" ]; then
+ echo Backup directory not found, aborting
+ exit 1
+ fi
+ if [ -f "/var/lib/ambari-server/resources/scripts/configs.py" ]; then
+ if [ -d "$AMBARI_CONFIG_DIR" ]; then
+ echo Restoring metron config from files in $AMBARI_CONFIG_DIR
+ i=0
+ for filename in $AMBARI_CONFIG_DIR/*;
+ do
+ [ -e "$filename" ] || continue
+ ((i=i+1))
+ filename=${filename##*/}
+ echo $i. Found config: $filename
+ config_type=${filename%.json}
+ echo " Setting config_type to $config_type"
+ /var/lib/ambari-server/resources/scripts/configs.py -u $username -p $password -a set -l ${ambari_address%:*} -n $cluster_name -c $config_type -f $AMBARI_CONFIG_DIR/${config_type}.json
+ echo " Done restoring $config_type"
+ done
+ echo Done restoring $i metron config files from $OUT_DIR
+ else
+ echo Ambari backup directory not found, skipping
+ fi
+ else
+ echo Skipping Ambari config restore - Ambari not found on this host
+ fi
+
+ if [ -f "$METRON_HOME/bin/zk_load_configs.sh" ]; then
+ if [ -d "$ZK_CONFIG_DIR" ]; then
+ echo Restoring Metron zookeeper config from files in $ZK_CONFIG_DIR
+ $METRON_HOME/bin/zk_load_configs.sh -m PUSH -i $ZK_CONFIG_DIR -z $ZOOKEEPER -f
+ echo Pulling config locally into Metron home config dir
+ $METRON_HOME/bin/zk_load_configs.sh -m PULL -o ${METRON_HOME}/config/zookeeper -z $ZOOKEEPER -f
+ echo Done restoring Metron zookeeper config from files in $ZK_CONFIG_DIR
+ else
+ echo Metron config backup directory not found, skipping
+ fi
+ else
+ echo Skipping Metron config restore - Metron not found on this host
+ fi
+else
+ echo Mode "$mode" not recognized. Exiting.
+fi