You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by jz...@apache.org on 2017/09/08 16:13:07 UTC
[opennlp-sandbox] branch master updated: OPENNLP-1111: Making tests
on EC2 automated.
This is an automated email from the ASF dual-hosted git repository.
jzemerick pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/opennlp-sandbox.git
The following commit(s) were added to refs/heads/master by this push:
new 9c78236 OPENNLP-1111: Making tests on EC2 automated.
9c78236 is described below
commit 9c78236bb5f422d4191b91a98ac2d412d23947bd
Author: jzonthemtn <jz...@apache.org>
AuthorDate: Fri Sep 8 12:10:44 2017 -0400
OPENNLP-1111: Making tests on EC2 automated.
---
aws-ec2-testing-scripts/README.md | 57 +++++++++++++++++-----
aws-ec2-testing-scripts/build-ami.sh | 4 ++
aws-ec2-testing-scripts/cf-template.json | 53 +++++++++++++--------
aws-ec2-testing-scripts/notify.sh | 25 ++++++----
aws-ec2-testing-scripts/packer.json | 60 ++++++++++++++++++++++++
aws-ec2-testing-scripts/run-eval-tests.sh | 3 +-
aws-ec2-testing-scripts/run-high-memory-tests.sh | 3 +-
7 files changed, 161 insertions(+), 44 deletions(-)
diff --git a/aws-ec2-testing-scripts/README.md b/aws-ec2-testing-scripts/README.md
index fb25a39..13f0ad6 100644
--- a/aws-ec2-testing-scripts/README.md
+++ b/aws-ec2-testing-scripts/README.md
@@ -1,26 +1,57 @@
# OpenNLP Testing Scripts
-These are scripts useful when testing OpenNLP builds on EC2.
+These are scripts to automate running OpenNLP's evaluation tests on AWS EC2. The scripts are composed of a Packer script, a CloudFormation template, and a few supporting bash scripts.
-## Directory Structure
+## Running OpenNLP Tests
-These scripts are written expecting the following directory structure:
+To run the tests two actions must be performed:
-* `/opt/` - Contains these scripts.
-* `/opt/opennlp` - Contains the OpenNLP code as cloned from https://github.com/apache/opennlp.
-* `/opt/opennlp-data` - Contains the data required for some of the OpenNLP tests. Contact dev@opennlp.apache.org for information on this data.
+1. Create an AMI that contains the required tools and OpenNLP test data.
+1. Launch a CloudFormation stack that creates an instance from the AMI and runs the tests.
+
+These two steps are described in detail below.
+
+### Creating the AMI used for Testing
+
+Creating the AMI requires the [Packer](https://www.packer.io/intro/index.html) tool. To create the AMI execute the `build-ami.sh` script. You may need to modify the location of the Packer executable in the `build-ami.sh` script. The OpenNLP test data should exist as `opennlp-data.zip` in the current directory prior to creating the AMI. This allows the Packer script to upload the test data to the instance when creating the AMI.
+
+You only need to create the AMI once. The same AMI can be reused for testing future OpenNLP versions. The only need to create a new AMI is to include updated OpenNLP test data.
+
+### Create the CloudFormation Stack
-## EC2 Instance Requirements
+Using the `cf-template.json` CloudFormation template create a new stack. The `Image` parameter should reference the AMI created by Packer. Be sure to check your email and confirm your subscription to the newly created SNS topic in order to receive the build emails.
-The instance defined in the CloudFormation template `cf-template.json` satisfies these requirements. You can create a stack from the template either through the AWS Console or using the AWS CLI.
+You can create a stack from the template either through the AWS Console or using the AWS CLI:
-* The instance must have the AWS CLI installed.
-* The scripts use SNS to send notifications so the instance must have permissions to publish SNS messages through either an instance role or via access/secret keys configured in the AWS CLI.
-* You must have an existing SNS topic configured to publish messages to and you must set the ARN in the `notify.sh` script.
+```
+aws cloudformation create-stack \
+ --stack-name OpenNLP-Testing \
+ --template-body file://./cf-template.json \
+ --parameters \
+ ParameterKey=InstanceType,ParameterValue=m4.xlarge \
+ ParameterKey=KeyName,ParameterValue=keyname \
+ ParameterKey=NotificationsEmail,ParameterValue=your@email.com \
+ ParameterKey=Branch,ParameterValue=opennlp-1.8.3 \
+ ParameterKey=Tests,ParameterValue=run-eval-tests.sh
+```
-## Notifications and Results
+When the tests are complete (either as success or failure) the email address specified in the `NotificationsEmail` parameter will receive an email notification. The email's subject will indicate if the tests were successful or failed and the email's body will contain approximately the last 200 KB of text from the Maven build log. Once you receive the notification you can terminate the stack or you can SSH into the EC2 instance if you need to debug or re-run any tests.
-You can configure the subject, message, and destination (topic ARN) in the `notify.sh` script. The build log will be too large (>256KB) for sending in the SNS message so SSH into the instance to check the log.
+```
+aws cloudformation delete-stack --stack-name OpenNLP-Testing
+```
+
+## AWS Infrastructure
+
+The `cf-template.json` CloudFormation template creates a new VPC to contain the EC2 instance that runs the tests. The template creates all the necessary components such as the route table, subnet, IAM policies and roles, and security group.
+
+### Instance Directory Structure
+
+These scripts are written expecting the following directory structure:
+
+* `/opt/` - Contains these scripts.
+* `/opt/opennlp` - Contains the OpenNLP code as cloned from https://github.com/apache/opennlp.
+* `/opt/opennlp-data` - Contains the data required for OpenNLP's eval tests.
## License
diff --git a/aws-ec2-testing-scripts/build-ami.sh b/aws-ec2-testing-scripts/build-ami.sh
new file mode 100755
index 0000000..fe887ef
--- /dev/null
+++ b/aws-ec2-testing-scripts/build-ami.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+packer build ./packer.json
+
diff --git a/aws-ec2-testing-scripts/cf-template.json b/aws-ec2-testing-scripts/cf-template.json
index 6df891b..2d8b3ca 100644
--- a/aws-ec2-testing-scripts/cf-template.json
+++ b/aws-ec2-testing-scripts/cf-template.json
@@ -8,20 +8,15 @@
"Default": "m4.xlarge"
},
"Image": {
- "Description": "The base AMI.",
+ "Description": "The OpenNLP testing AMI made with the Packer script.",
"Type": "String",
- "Default": "ami-80861296"
+ "Default": "ami-6191821a"
},
"KeyName": {
"Description": "An existing EC2 keypair.",
"Type": "AWS::EC2::KeyPair::KeyName",
"ConstraintDescription": "Must be the name of an existing EC2 keypair."
},
- "OpenNLPDataOnS3": {
- "Description": "The location of the OpenNLP evaluation test data on S3.",
- "Type": "String",
- "ConstraintDescription": "s3://bucket-name/opennlp-data.zip"
- },
"NotificationsEmail": {
"Description": "Email address to receive notifications.",
"Type": "String"
@@ -30,6 +25,17 @@
"Description": "IP to allow SSH.",
"Type": "String",
"Default": "0.0.0.0/0"
+ },
+ "Branch": {
+ "Description": "The OpenNLP git branch or tag to test.",
+ "Type": "String",
+ "Default": "opennlp-1.8.2"
+ },
+ "Tests": {
+ "Description": "The OpenNLP tests to run.",
+ "Type": "String",
+ "AllowedValues": ["run-eval-tests.sh", "run-high-memory-tests.sh"],
+ "Default": "run-eval-tests.sh"
}
},
"Resources": {
@@ -221,6 +227,19 @@
},
"Protocol": "email"
}
+ ]
+ }
+ },
+ "SNSTopic": {
+ "Type": "AWS::SNS::Topic",
+ "Properties": {
+ "Subscription": [
+ {
+ "Endpoint": {
+ "Ref": "NotificationsEmail"
+ },
+ "Protocol": "email"
+ }
],
"TopicName": "OpenNLP-Notification"
}
@@ -267,21 +286,17 @@
"",
[
"#!/bin/bash -xe\n",
- "apt-get update && sudo apt-get -y dist-upgrade\n",
- "apt-get install -y openjdk-8-jdk git maven awscli\n",
- "# Get the scripts\n",
- "git clone https://github.com/apache/opennlp-sandbox.git\n",
- "mv opennlp-sandbox/aws-ec2-testing-scripts/* /opt/\n",
- "# Get OpenNLP\n",
+ "# Clone OpenNLP.\n",
"git clone https://github.com/apache/opennlp.git\n",
"mv opennlp /opt/\n",
+ "chown ubuntu:ubuntu /opt/ -R\n",
+ "# Checkout the branch or tag that we want to test.\n",
+ "cd /opt/opennlp\n",
+ "git checkout ", {"Ref": "Branch"}, "\n",
"sed -i 's/TOPICARNPARAM/", {"Ref": "SNSTopic"}, "/g' /opt/notify.sh\n",
- "mkdir /opt/opennlp-data\n",
- "aws s3 cp ",
- {
- "Ref": "OpenNLPDataOnS3"
- },
- " /opt/opennlp-data --recursive\n"
+ "# Start the tests\n",
+ "cd /opt\n",
+ "./", {"Ref": "Tests"}, "\n"
]
]
}
diff --git a/aws-ec2-testing-scripts/notify.sh b/aws-ec2-testing-scripts/notify.sh
index 1de6148..963f6f9 100755
--- a/aws-ec2-testing-scripts/notify.sh
+++ b/aws-ec2-testing-scripts/notify.sh
@@ -15,15 +15,24 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# Configure these values as desired.
-
-# Notifications via SNS.
+SUBJECT=$1
TOPIC_ARN="TOPICARNPARAM"
-SUBJECT="OpenNLP Notification"
+LOG_FILE="/opt/build.log"
+
+# The max size for a SNS body is 256KB.
+# We'll round down a bit to safely stay under that limit.
+tail -c 200000 $LOG_FILE > /tmp/subset.log
+
+OUTCOME="SUCCESS"
-# Received from the training scripts.
-MESSAGE=$1
-ACTION=$2
+# Look to see if the build failed.
+if grep -q 'BUILD FAILURE' "$LOG_FILE"; then
+ OUTCOME="FAILURE"
+fi
# Publish the message to SNS.
-aws sns publish --topic-arn "$TOPIC_ARN" --message "$MESSAGE" --subject "$SUBJECT"
+aws sns publish \
+ --region us-east-1 \
+ --topic-arn "$TOPIC_ARN" \
+ --subject "$OUTCOME - $SUBJECT" \
+ --message file:///tmp/subset.log
diff --git a/aws-ec2-testing-scripts/packer.json b/aws-ec2-testing-scripts/packer.json
new file mode 100644
index 0000000..945f77f
--- /dev/null
+++ b/aws-ec2-testing-scripts/packer.json
@@ -0,0 +1,60 @@
+{
+ "variables": {},
+ "builders": [
+ {
+ "type": "amazon-ebs",
+ "region": "us-east-1",
+ "source_ami": "ami-cd0f5cb6",
+ "instance_type": "m3.medium",
+ "ssh_username": "ubuntu",
+ "ami_name": "OpenNLP Testing {{timestamp}}",
+ "tags": {
+ "Name": "OpenNLP Testing"
+ }
+ }
+ ],
+ "provisioners": [
+ {
+ "type": "file",
+ "source": "notify.sh",
+ "destination": "/tmp/"
+ },
+ {
+ "type": "file",
+ "source": "run-eval-tests.sh",
+ "destination": "/tmp/"
+ },
+ {
+ "type": "file",
+ "source": "run-high-memory-tests.sh",
+ "destination": "/tmp/"
+ },
+ {
+ "type": "file",
+ "source": "opennlp-data.zip",
+ "destination": "/tmp/"
+ },
+ {
+ "type": "shell",
+ "inline": [
+ "sudo apt-get update",
+ "sudo apt-get install -y openjdk-8-jdk maven git awscli unzip",
+ "sudo mv /tmp/*.sh /opt/",
+ "sudo chown ubuntu:ubuntu /opt/*.sh",
+ "sudo chmod +x /opt/*.sh",
+ "sudo mkdir /opt/opennlp-data",
+ "sudo chown ubuntu:ubuntu /opt/opennlp-data",
+ "unzip /tmp/opennlp-data.zip -d /opt/opennlp-data",
+ "tar -xzf /opt/opennlp-data/ontonotes4/data.tar.gz -C /opt/opennlp-data/ontonotes4/",
+ "sudo sed -i 's/PermitRootLogin without-password/PermitRootLogin forced-commands-only/g' /etc/ssh/sshd_config",
+ "sudo passwd -l root",
+ "sudo shred -n 50 -fuzv /etc/ssh/*_key /etc/ssh/*_key.pub",
+ "sudo find /root/.ssh -type f -exec shred -n 30 -z -u {} \\;",
+ "sudo find /home/ubuntu/.ssh -type f -exec shred -n 30 -z -u {} \\;",
+ "sudo shred -n 50 -fuzv /var/log/wtmp",
+ "sudo shred -n 50 -fuzv /var/log/btmp",
+ "sudo shred -n 50 -fuzv /var/log/lastlog"
+ ]
+ }
+ ]
+}
diff --git a/aws-ec2-testing-scripts/run-eval-tests.sh b/aws-ec2-testing-scripts/run-eval-tests.sh
index b98bd1e..2d44456 100755
--- a/aws-ec2-testing-scripts/run-eval-tests.sh
+++ b/aws-ec2-testing-scripts/run-eval-tests.sh
@@ -15,5 +15,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-rm -f ./nohup.out
-nohup sh -c 'cd opennlp && mvn clean install -Peval-tests -DOPENNLP_DATA_DIR=/opt/opennlp-data/ && /opt/notify.sh "eval-tests complete" "eval-tests"' &
+nohup sh -c 'cd /opt/opennlp && mvn clean install -l /opt/build.log -Peval-tests -DOPENNLP_DATA_DIR=/opt/opennlp-data/ && /opt/notify.sh "OpenNLP eval-tests complete"' &
diff --git a/aws-ec2-testing-scripts/run-high-memory-tests.sh b/aws-ec2-testing-scripts/run-high-memory-tests.sh
index 3891318..fbd0066 100755
--- a/aws-ec2-testing-scripts/run-high-memory-tests.sh
+++ b/aws-ec2-testing-scripts/run-high-memory-tests.sh
@@ -15,5 +15,4 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-rm -f nohup.out
-nohup sh -c 'cd opennlp && mvn clean install -Phigh-memory-tests -DOPENNLP_DATA_DIR=/opt/opennlp-data/ && /opt/notify.sh "High memory tests complete" "high-memory-tests"' &
+nohup sh -c 'cd /opt/opennlp && mvn clean install -l /opt/build.log -Phigh-memory-tests -DOPENNLP_DATA_DIR=/opt/opennlp-data/ && /opt/notify.sh "OpenNLP High-memory tests complete"' &
--
To stop receiving notification emails like this one, please contact
['"commits@opennlp.apache.org" <co...@opennlp.apache.org>'].