You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ni...@apache.org on 2016/05/02 21:12:50 UTC
incubator-metron git commit: METRON-128 Create Platform Launch and
Troubleshooting Scripts (nickwallen) closes apache/incubator-metron#98
Repository: incubator-metron
Updated Branches:
refs/heads/master 645d8292b -> dc8a14e22
METRON-128 Create Platform Launch and Troubleshooting Scripts (nickwallen) closes apache/incubator-metron#98
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/dc8a14e2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/dc8a14e2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/dc8a14e2
Branch: refs/heads/master
Commit: dc8a14e225361cbf93f86e6b1cad11194679355b
Parents: 645d829
Author: nickwallen <ni...@nickallen.org>
Authored: Mon May 2 15:11:54 2016 -0400
Committer: Nick Allen <ni...@nickallen.org>
Committed: Mon May 2 15:11:54 2016 -0400
----------------------------------------------------------------------
metron-deployment/amazon-ec2/.gitignore | 1 +
metron-deployment/amazon-ec2/README.md | 52 ++++++++++++------
metron-deployment/amazon-ec2/run.sh | 70 +++++++++++++++++++++++++
metron-deployment/scripts/platform-info.sh | 64 ++++++++++++++++++++++
4 files changed, 171 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/dc8a14e2/metron-deployment/amazon-ec2/.gitignore
----------------------------------------------------------------------
diff --git a/metron-deployment/amazon-ec2/.gitignore b/metron-deployment/amazon-ec2/.gitignore
index 9c214d2..34c24d2 100644
--- a/metron-deployment/amazon-ec2/.gitignore
+++ b/metron-deployment/amazon-ec2/.gitignore
@@ -2,3 +2,4 @@
*.secret
*.log
*.retry
+.metron-env
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/dc8a14e2/metron-deployment/amazon-ec2/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/amazon-ec2/README.md b/metron-deployment/amazon-ec2/README.md
index 90f6fd0..6fd6326 100644
--- a/metron-deployment/amazon-ec2/README.md
+++ b/metron-deployment/amazon-ec2/README.md
@@ -3,18 +3,40 @@ Apache Metron on Amazon EC2
This project fully automates the provisioning of Apache Metron on Amazon EC2 infrastructure. Starting with only your Amazon EC2 credentials, this project will create a fully-functioning, end-to-end, multi-node cluster running Apache Metron.
+Warning: Amazon will charge for the use of their resources when running Apache Metron. The amount will vary based on the number and size of hosts, along with current Amazon pricing structure. Be sure to stop or terminate all of the hosts instantiated by Apache Metron when not in use to avoid unnecessary charges.
+
Getting Started
---------------
### Prerequisites
-The host that will drive the provisioning process will need to have [Ansible](https://github.com/ansible/ansible), Python and PIP installed. In most cases, a development laptop serves this purpose just fine. Also, install the Python library `boto` and its dependencies.
+The computer used to deploy Apache Metron will need to have [Ansible](https://github.com/ansible/ansible), Python, Maven, SSH, and Git installed. Any platform that supports these tools is suitable, but the following instructions cover only Mac OS X. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.
-```
-pip install boto six
-```
+1. Install Homebrew by running the following command in a terminal. Refer to the [Homebrew](http://brew.sh/) home page for the latest installation instructions.
+
+ ```
+ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
+ ```
+
+2. With Homebrew installed, run the following command in a terminal to install all of the required tools.
+
+ ```
+ brew install ansible brew-pip maven git
+ ```
+
+3. Ensure that a public SSH key is located at `~/.ssh/id_rsa.pub`.
+
+ ```
+ $ cat ~/.ssh/id_rsa.pub
+ ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChv5GJxPjR39UJV7VY17ivbLVlxFrH7UHwh1Jsjem4d1eYiAtde5N2y65/HRNxWbhYli9ED8k0/MRP92ejewucEbrPNq5mytPqdC4IvZ98Ln2GbqTDwvlP3T7xa/wYFOpFsOmXXql8216wSrnrS4f3XK7ze34S6/VmY+lsBYnr3dzyj8sG/mexpJgFS/w83mWJV0e/ryf4Hd7P6DZ5fO+nmTXfKNK22ga4ctcnbZ+toYcPL+ODCh8598XCKVo97XjwF5OxN3vl1p1HHguo3cHB4H1OIaqX5mUt59gFIZcAXUME89PO6NUiZDd3RTstpf125nQVkQAHu2fvW96/f037 nick@localhost
+ ```
+
+ If this file does not exist, run the following command at a terminal and accept all defaults. Only the public key, not the private key, will be uploaded to Amazon and configured on each host to enable SSH connectivity. While it is possible to create and use an alternative key those details will not be covered.
+
+ ```
+ ssh-keygen -t rsa
+ ```
-Ensure that an SSH key has been generated and stored at `~/.ssh/id_rsa.pub`. In most cases this key will already exist and no further action will be needed.
### Create User
@@ -29,28 +51,26 @@ Ensure that an SSH key has been generated and stored at `~/.ssh/id_rsa.pub`. In
3. Create an access key for the user by clicking on `Security Credentials > Create Access Key`. Save the provided access key values in a safe place. These values cannot be retrieved from the web console at a later time.
-4. Use the access key by exporting its values to the shell's environment. This allows Ansible to authenticate with Amazon EC2. For example:
+### Deploy Metron
+
+1. Use the Amazon access key by exporting its values via the shell's environment. This allows Ansible to authenticate with Amazon EC2. For example:
```
export AWS_ACCESS_KEY_ID="AKIAI6NRFEO27E5FFELQ"
export AWS_SECRET_ACCESS_KEY="vTDydWJQnAer7OWauUS150i+9Np7hfCXrrVVP6ed"
```
-### Deploy Metron
+ Notice: You must replace the access key values above with values from your own access key.
-1. Ensure that Metron's streaming topology uber-jar has been built.
+2. Start the Apache Metron deployment process. When prompted provide a unique name for your Metron environment or accept the default.
```
- cd ../../metron-platform
- mvn clean package -DskipTests
+ $ ./run.sh
+ Metron Environment [metron-test]: my-metron-env
+ ...
```
-2. Start the Metron playbook. A full Metron deployment can consume up to 60 minutes. Grab a coffee, relax and practice mindfulness meditation. If the playbook fails mid-stream for any reason, simply re-run it.
-
- ```
- export EC2_INI_PATH=conf/ec2.ini
- ansible-playbook -i ec2.py playbook.yml
- ```
+ The process is likely to take between 70-90 minutes. Fortunately, everything is fully automated and you should feel free to grab a coffee.
### Explore Metron
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/dc8a14e2/metron-deployment/amazon-ec2/run.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/amazon-ec2/run.sh b/metron-deployment/amazon-ec2/run.sh
new file mode 100755
index 0000000..dd7f86a
--- /dev/null
+++ b/metron-deployment/amazon-ec2/run.sh
@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+# Builds Metron platform jars, instantiates hosts, and deploys Metron to those
+# hosts on Amazon EC2
+#
+
+LOGFILE="./ansible.log"
+DEPLOYDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+EXTRA_VARS="-v"
+NOW=`date`
+DEFAULT_ENV="metron-test"
+DEFAULT_ENV_FILE="./.metron-env"
+
+# ensure aws access key is defined
+if [ -z "$AWS_ACCESS_KEY_ID" ]; then
+ echo "Error: missing AWS_ACCESS_KEY_ID"
+ exit 1
+fi
+
+# ensure aws access key is defined
+if [ -z "$AWS_SECRET_ACCESS_KEY" ]; then
+ echo "Error: missing AWS_SECRET_ACCESS_KEY"
+ exit 1
+fi
+
+# retrieve environment name from previous run
+if [ -f $DEFAULT_ENV_FILE ]; then
+ ENV=`cat $DEFAULT_ENV_FILE`
+else
+ ENV=$DEFAULT_ENV
+fi
+
+# prompt the user for an environment name
+read -p "Metron Environment [$ENV]: " INPUT
+[ -n "$INPUT" ] && ENV=$INPUT
+
+# store the environment name for the next run
+echo "$ENV" > $DEFAULT_ENV_FILE
+
+# log information about the host platform
+echo "=============================================================" >> $LOGFILE
+echo "Launching Metron @ $NOW"... >> $LOGFILE
+echo "Metron Environment: $ENV" >> $LOGFILE
+$DEPLOYDIR/../scripts/platform-info.sh >> $LOGFILE
+
+# build metron
+cd ../../metron-platform
+mvn package -DskipTests
+
+# deploy metron
+cd $DEPLOYDIR
+export EC2_INI_PATH=conf/ec2.ini
+ansible-playbook -i ec2.py playbook.yml --extra-vars="env=$ENV" $EXTRA_VARS
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/dc8a14e2/metron-deployment/scripts/platform-info.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/scripts/platform-info.sh b/metron-deployment/scripts/platform-info.sh
new file mode 100755
index 0000000..be53e86
--- /dev/null
+++ b/metron-deployment/scripts/platform-info.sh
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+# extracts information from the host environment that is useful for
+# troubleshooting Apache Metron deployments
+#
+CWD="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+# extract metron version from the pom
+METRON_VERSION=`cat $CWD/../../pom.xml | grep "<version>" | head -1 | sed -ne '/version/{s/.*<version>\(.*\)<\/version>.*/\1/p;q;}'`
+echo "Metron $METRON_VERSION"
+
+# is this a git repo?
+IS_GIT_REPO=`git rev-parse --is-inside-work-tree`
+if [ "$IS_GIT_REPO" == "true" ]; then
+
+ # current branch
+ echo "--"
+ git branch | grep "*"
+
+ # last commit
+ echo "--"
+ git log -n 1
+
+ # local changes since last commit
+ echo "--"
+ git diff --stat
+fi
+
+# ansible
+echo "--"
+ansible --version
+
+# vagrant
+echo "--"
+vagrant --version
+
+# python
+echo "--"
+python --version 2>&1
+
+# maven
+echo "--"
+mvn --version
+
+# operating system
+echo "--"
+uname -a