[GitHub] [accumulo-testing] dlmarion commented on a change in pull request #185: Terraform configurations for creating AWS test infrastructure

[GitBox <gi...@apache.org>]

dlmarion commented on a change in pull request #185:
URL: https://github.com/apache/accumulo-testing/pull/185#discussion_r780421693



##########
File path: contrib/terraform-testing-infrastructure/README.md
##########
@@ -0,0 +1,227 @@
+# Accumulo Testing Infrastructure
+
+## Description
+
+This Git repository contains several [Terraform](https://www.terraform.io/) configurations.
+
+  - `shared_state` creates an AWS S3 Bucket and DynamoDB table that are a prerequisite for the Terraform configuration in `aws`.
+  - `aws` creates the following AWS resources:
+    1. Creates an EFS filesystem in AWS (think NFS). On this filesystem we are going to install Maven, ZooKeeper, Hadoop, Accumulo, and Accumulo Testing
+    2. Creates IP addresses for the EFS filesystem in each AWS availability zone that we use. The IP address allows us to mount the EFS filesystem to EC2 nodes.
+    3. Creates one or more EC2 nodes for running the different components. Currently the configuration uses the m5.2xlarge instance type which provides 8 vCPUs, 32GB RAM, and an EBS backed root volume.
+    4. Runs commands on the EC2 nodes after they are started (5 minutes according to the docs) to install software and configure them.
+    5. Creates DNS A records for the EC2 nodes.
+
+## Prerequisites
+
+You will need to download and install the correct Terraform [CLI](https://www.terraform.io/downloads) for your platform. Put the `terraform` binary on your PATH.
+
+## Shared State
+
+The `shared_state` directory contains a Terraform configuration for creating an S3 bucket and DynamoDB table. These objects only need to be created once and are used for sharing the Terraform state with a team. To read more about this see [remote state](https://www.terraform.io/docs/language/state/remote.html). I used [these](https://blog.gruntwork.io/how-to-manage-terraform-state-28f5697e68fa) instructions for creating these objects. This configuration is simple, there are no variables, but you may need to change the aws region in `main.tf`. If you change the name of the S3 bucket or DynamoDB table, then you will need to update the `main.tf` file in the `aws` directory. To create the S3 bucket and DynamoDB table, run `terraform init`, followed by `terraform apply`.
+
+## Test Cluster
+
+The `aws` directory contains a Terrform configuration and related files for creating an Accumulo cluster on AWS infrastructure. This configuration consists of the following items in this directory:
+
+  - main.tf - the Terraform configuration file
+  - variables.tf - the declaration and default values for variables
+  - templates/ - template files that are processed during the `apply` operation
+  - files/ - non-template files that are uploaded to the cluster during the `apply` operation
+  - conf/ - a non-git tracked directory that contains the rendered template files that are uploaded to the cluster
+
+### Variables
+
+The table below lists the variables used in this configuration and their default values.
+
+| Variable Name | Default Value | Required |
+|---------------|---------------|----------|
+| instance_count | 2 | true |
+| instance_type | m5.2xlarge | true |
+| root_volume_gb | 300 | true |
+| efs_mount | /efs | true |
+| security_group | | true |
+| us_east_1b_subnet | |  true |
+| us_east_1e_subnet | |  true |
+| route53_zone | |  true |
+| ami_owner | |  true |
+| ami_name_pattern | |  true |
+| authorized_ssh_keys | | true |
+| zookeeper_dir | /data/zookeeper | true |
+| hadoop_dir | /data/hadoop | true |
+| accumulo_dir | /data/accumulo | true |
+| maven_version | 3.8.4 |  true |
+| zookeeper_version | 3.5.9 | true |
+| hadoop_version | 3.3.1 | true |
+| accumulo_version | 2.1.0-SNAPSHOT | true |
+| accumulo_repo | https://github.com/apache/accumulo.git | true |
+| accumulo_branch_name | main | true |
+| accumulo_testing_repo | https://github.com/apache/accumulo-testing.git | true |
+| accumulo_testing_branch_name | main | true |
+| local_sources_dir | | false |
+
+### Configuration
+
+You will need to provide values for the required variables with no default value. You can also override the default values by supplying new values. There are several [ways](https://www.terraform.io/language/values/variables#assigning-values-to-root-module-variables) to do this. Below is an example of a JSON file that contains values, which can be placed in the `aws` directory with a name that ends with `.auto.tfvars.json`, that will be applied during the `terraform apply` command. 
+
+```
+{
+  "security_group": "sg-ABCDEF001",
+  "route53_zone": "some.domain.com",
+  "us_east_1b_subnet": "subnet-ABCDEF123",
+  "us_east_1e_subnet": "subnet-ABCDEF124",
+  "ami_owner": "000000000001",
+  "ami_name_pattern": "MY_AMI_*",
+  "authorized_ssh_keys": [
+    "ssh-rsa dev_key_1",
+    "ssh-rsa dev_key_2"
+  ]
+}
+```
+
+## AWS Resources
+
+This Terraform configuration creates:
+
+  1. An EFS filesystem with IP addresses in the `${us_east_1b_subnet}` and `${us_east_1e_subnet}` subnets
+  2. `${instance_count}` EC2 nodes of `${instance_type}` with the latest AMI matching `${ami_name_pattern}` from the `${ami_owner}`. Each EC2 node will have a `${root_volume_gb}`GB root volume. The EFS filesystem is NFS mounted to each node at `${efs_mount}`.
+  3. DNS entries in Route53 for each EC2 node.
+
+## Software Layout
+
+This Terraform configuration:
+
+  1. Downloads, if necessary, the Apache Maven `${maven_version}` binary tarball to `${efs_mount}/sources`, then untars it to `${efs_mount}/apache-maven/apache-maven-${maven_version}`
+  2. Downloads, if necessary, the Apache Zookeeper `${zookeer_version}` binary tarball to `${efs_mount}/sources`, then untars it to `${efs_mount}/zookeeper/apache-zookeeper-${zookeeper_version}-bin`
+  3. Downloads, if necessary, the Apache Hadoop `${hadoop_version}` binary tarball to `${efs_mount}/sources`, then untars it to `${efs_mount}/hadoop/hadoop-${hadoop_version}`
+  4. Downloads, if necessary, clones the Apache Accumulo Git repo from `${accumulo_repo}` into `${efs_mount}/sources/accumulo-repo`. It switches to the `${accumulo_branch_name}` branch and builds the software using Maven, then untars the binary tarball to `${efs_mount}/accumulo/accumulo-${accumulo_version}`

Review comment:
       Good catch. An earlier version download the `accumulo` and `accumulo-testing` repos vs cloning them.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org