You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@couchdb.apache.org by da...@apache.org on 2019/12/10 19:21:55 UTC
[couchdb-infra-cm] branch main created (now 731dd9c)
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb-infra-cm.git.
at 731dd9c Add new roles and provision playbooks
This branch includes the following new commits:
new 3d202ef Initial commit
new a329b17 Basic inventory generation
new 731dd9c Add new roles and provision playbooks
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[couchdb-infra-cm] 03/03: Add new roles and provision playbooks
Posted by da...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb-infra-cm.git
commit 731dd9c716fc552145f80627cb1b42b85d961983
Author: Paul J. Davis <pa...@gmail.com>
AuthorDate: Wed Aug 14 14:09:34 2019 -0500
Add new roles and provision playbooks
---
README.md | 17 +++++++++++++
ci_agents.yml | 5 ++++
group_vars/all.yaml | 1 +
provision.yml | 30 +++++++++++++++++++++++
roles/ci_agent/tasks/main.yml | 55 +++++++++++++++++++++++++++++++++++++++++++
roles/common/tasks/main.yml | 29 +++++++++++++++++++++++
6 files changed, 137 insertions(+)
diff --git a/README.md b/README.md
index 82a2d52..3eac5aa 100644
--- a/README.md
+++ b/README.md
@@ -20,3 +20,20 @@ Create a `~/.couchdb-infra-cm.cfg` file that contains the following options:
api_key = <REDACTED>
Then simply run the script which will dump the current inventory to stdout. Redirect the output to whatever filename you so desire.
+
+Provisioning a Node
+---
+
+First, create a new VM of the desired type using whatever means necessary to have root SSH access along with a public IP address (or at least, some method that can be configured into Ansible though you're on your own at this point).
+
+Then run:
+
+ ansible-playbook -i W.X.Y.Z, provision.yml
+
+*Note:* Make sure to include the trailing comma (,) in the -i argument or you'll get an error about not being able to parse the inventory.
+
+Once this has run and you have updated the `production` inventory file (See the section above on generating inventory files) in this directory you can then run:
+
+ ansible-playbook -i production ci_agents.yml
+
+And the node will be configured as a new CI agent.
\ No newline at end of file
diff --git a/ci_agents.yml b/ci_agents.yml
new file mode 100644
index 0000000..8278f40
--- /dev/null
+++ b/ci_agents.yml
@@ -0,0 +1,5 @@
+---
+- hosts: ci_agents
+ roles:
+ - common
+ - ci_agent
\ No newline at end of file
diff --git a/group_vars/all.yaml b/group_vars/all.yaml
new file mode 100644
index 0000000..3e9e888
--- /dev/null
+++ b/group_vars/all.yaml
@@ -0,0 +1 @@
+ansible_ssh_user: couchdb
diff --git a/provision.yml b/provision.yml
new file mode 100644
index 0000000..24e54af
--- /dev/null
+++ b/provision.yml
@@ -0,0 +1,30 @@
+---
+- hosts: all
+ vars:
+ ansible_ssh_user: root
+ tasks:
+ - name: Add user couchdb
+ user:
+ name: couchdb
+ state: present
+ shell: /bin/bash
+
+ - name: Make sure sudo is installed
+ apt:
+ name: sudo
+ state: latest
+
+ - name: Grant sudo access to couchdb user
+ copy:
+ content: 'couchdb ALL=(ALL) NOPASSWD:ALL'
+ dest: /etc/sudoers.d/couchdb
+ mode: 0440
+
+ - name: Allow CouchDB Infra access to couchdb user
+ authorized_key:
+ user: couchdb
+ state: present
+ key: "{{ item }}"
+ with_items:
+ - https://github.com/davisp.keys
+ - https://github.com/wohali.keys
\ No newline at end of file
diff --git a/roles/ci_agent/tasks/main.yml b/roles/ci_agent/tasks/main.yml
new file mode 100644
index 0000000..edf6791
--- /dev/null
+++ b/roles/ci_agent/tasks/main.yml
@@ -0,0 +1,55 @@
+- name: Install Docker gpg key for Apt
+ become: yes
+ apt_key:
+ url: https://download.docker.com/linux/debian/gpg
+ state: present
+
+- name: Setup Docker Apt repository
+ become: yes
+ apt_repository:
+ repo: deb https://download.docker.com/linux/debian {{ ansible_distribution_release }} stable
+ filename: docker
+ state: present
+
+- name: Install Docker Packages
+ become: yes
+ apt:
+ name: "{{ packages }}"
+ vars:
+ packages:
+ - containerd.io
+ - docker-ce
+ - docker-ce-cli
+
+- name: Install multi-architecture support for Docker
+ become: yes
+ apt:
+ name: "{{ packages }}"
+ state: latest
+ vars:
+ packages:
+ - binfmt-support
+ - qemu
+ - qemu-user-static
+
+- name: Install Java 8
+ become: yes
+ apt:
+ name: "{{ packages }}"
+ state: latest
+ vars:
+ packages:
+ - openjdk-8-jre-headless
+
+# Copy infra script to delete old docker images
+# Add cron to run docker deleter script
+
+# Add jenkins user with specific uid
+# Write script to download a jar and run it
+# Install script as runit service
+
+# Figure out how to configure the password things required
+# for running the jar bits
+
+# Figure out if our Java agent requires all the jars
+# and tooling I saw in the docs (ant, maven, multiple jvms, etc)
diff --git a/roles/common/tasks/main.yml b/roles/common/tasks/main.yml
new file mode 100644
index 0000000..edd83cc
--- /dev/null
+++ b/roles/common/tasks/main.yml
@@ -0,0 +1,29 @@
+- name: Install basic ubiquitous packages
+ become: yes
+ apt:
+ name: "{{ packages }}"
+ state: latest
+ update_cache: yes
+ cache_valid_time: 3600
+ vars:
+ packages:
+ - apt-transport-https
+ - atop
+ - ca-certificates
+ - curl
+ - git
+ - gnupg2
+ - iftop
+ - iperf3
+ - mtr-tiny
+ - nload
+ - python3
+ - runit
+ - runit-systemd
+ - screen
+ - software-properties-common
+ - strace
+ - tcpdump
+ - tmux
+ - vim
+ - wget
[couchdb-infra-cm] 01/03: Initial commit
Posted by da...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb-infra-cm.git
commit 3d202efb9aae300e5b276daad8abb4daeb05a1cd
Author: Paul J. Davis <pa...@gmail.com>
AuthorDate: Fri Aug 9 13:28:55 2019 -0500
Initial commit
---
.gitignore | 1 +
README.md | 11 +++++++++++
requirements.txt | 1 +
3 files changed, 13 insertions(+)
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..5ceb386
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1 @@
+venv
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..fefe37b
--- /dev/null
+++ b/README.md
@@ -0,0 +1,11 @@
+CouchDB Infrastructure Config Management
+===
+
+This repository contains Ansible scripts for managing our VM testing infrastructure.
+
+Setup
+---
+
+ $ virtualenv venv
+ $ source venv/bin/activate
+ $ pip install -r requirements.txt
diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 0000000..90d4055
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1 @@
+ansible
[couchdb-infra-cm] 02/03: Basic inventory generation
Posted by da...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
davisp pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb-infra-cm.git
commit a329b177223ab247ce936f2de4edcb42725490fa
Author: Paul J. Davis <pa...@gmail.com>
AuthorDate: Fri Aug 9 14:50:19 2019 -0500
Basic inventory generation
For the time being I'm relying on a tool to generate our inventory
lists. In the future if we get our permissions scheme setup we'll be
able to open this up to anyone that has an IBM Cloud account that is
associated with the CouchDB CI Infra account.
---
README.md | 11 ++++
production | 24 +++++++++
requirements.txt | 1 +
tools/gen-inventory | 152 ++++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 188 insertions(+)
diff --git a/README.md b/README.md
index fefe37b..82a2d52 100644
--- a/README.md
+++ b/README.md
@@ -9,3 +9,14 @@ Setup
$ virtualenv venv
$ source venv/bin/activate
$ pip install -r requirements.txt
+
+
+Generating Inventory Listings
+---
+
+Create a `~/.couchdb-infra-cm.cfg` file that contains the following options:
+
+ [ibmcloud]
+ api_key = <REDACTED>
+
+Then simply run the script which will dump the current inventory to stdout. Redirect the output to whatever filename you so desire.
diff --git a/production b/production
new file mode 100644
index 0000000..940d582
--- /dev/null
+++ b/production
@@ -0,0 +1,24 @@
+all:
+ children:
+ ci_agents:
+ hosts:
+ 169.61.160.207:
+ boot_volume:
+ device: xvda
+ name: ci-farm-server-1-boot
+ instance:
+ created_at: '2019-08-09T17:10:36.149Z'
+ id: ff9bd14d-c0ea-4f99-9d62-0c9b2f566264
+ name: ci-farm-server-1
+ profile: cc1-4x8
+ subnet: couchdb-ci-farm-subnet-2
+ vpc: couchdb-ci-farm
+ zone: us-south-2
+ ips:
+ private: 10.240.64.4
+ public: 169.61.160.207
+ system:
+ arch: amd64
+ num_cpus: 4
+ ram: 8
+
diff --git a/requirements.txt b/requirements.txt
index 90d4055..8bf65d1 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1 +1,2 @@
ansible
+requests
diff --git a/tools/gen-inventory b/tools/gen-inventory
new file mode 100755
index 0000000..3f85b2a
--- /dev/null
+++ b/tools/gen-inventory
@@ -0,0 +1,152 @@
+#!/usr/bin/env python
+
+import ConfigParser as cp
+import json
+import os
+import re
+
+import requests
+import yaml
+
+
+IBM_CLOUD_URL = "https://us-south.iaas.cloud.ibm.com/v1/"
+IAM_URL = "https://iam.cloud.ibm.com/identity/token"
+
+IBM_CLOUD_GENERATION = "1"
+IBM_CLOUD_VERSION = "2019-08-09"
+
+API_KEY = None
+IAM_TOKEN = None
+SESS = requests.session()
+
+CI_AGENT_RE = re.compile("ci-farm-server-\d+")
+
+
+def tostr(obj):
+ ret = {}
+ for k, v in obj.items():
+ if isinstance(k, unicode):
+ k = k.encode("utf-8")
+ if isinstance(v, dict):
+ ret[k] = tostr(v)
+ elif isinstance(v, unicode):
+ ret[k] = v.encode("utf-8")
+ else:
+ ret[k] = v
+ return ret
+
+
+def load_api_key():
+ global API_KEY
+ path = os.path.expanduser("~/.couchdb-infra-cm.cfg")
+ if not os.path.exists(path):
+ print "Missing config file: " + path
+ exit(1)
+ parser = cp.SafeConfigParser()
+ parser.read([path])
+ API_KEY = parser.get("ibmcloud", "api_key")
+
+
+def load_iam_token():
+ global IAM_TOKEN
+ headers = {
+ "Accept": "application/json"
+ }
+ data = {
+ "grant_type": "urn:ibm:params:oauth:grant-type:apikey",
+ "apikey": API_KEY
+ }
+ resp = SESS.post(IAM_URL, headers=headers, data=data)
+ resp.raise_for_status()
+ body = resp.json()
+ IAM_TOKEN = body["token_type"] + " " + body["access_token"]
+ SESS.headers["Authorization"] = IAM_TOKEN
+
+
+def init():
+ load_api_key()
+ load_iam_token()
+
+
+def list_instances():
+ url = IBM_CLOUD_URL + "/instances"
+ params = {
+ "version": IBM_CLOUD_VERSION,
+ "generation": IBM_CLOUD_GENERATION,
+ "limit": 100
+ }
+ while url:
+ resp = SESS.get(url, params=params)
+ body = resp.json()
+ for instance in body["instances"]:
+ interface_url = instance["primary_network_interface"]["href"]
+ resp = SESS.get(interface_url, params=params)
+ instance["primary_network_interface"] = resp.json()
+ yield instance
+ url = body.get("next")
+
+
+def load_ci_agent(ci_agents, instance):
+ if instance["status"] != "running":
+ return
+
+ name = instance["name"]
+ floating_ips = instance["primary_network_interface"]["floating_ips"]
+ if not len(floating_ips):
+ print "Instance {} does not have any floating ips".format(name)
+ exit(2)
+
+ public_ip = floating_ips[0]["address"]
+
+ ci_agents[public_ip] = {
+ "instance": {
+ "id": instance["id"],
+ "name": instance["name"],
+ "created_at": instance["created_at"],
+ "profile": instance["profile"]["name"],
+ "vpc": instance["vpc"]["name"],
+ "zone": instance["zone"]["name"],
+ "subnet": instance["primary_network_interface"]["subnet"]["name"]
+ },
+ "ips": {
+ "public": public_ip,
+ "private": instance["primary_network_interface"]["primary_ipv4_address"]
+ },
+ "boot_volume": {
+ "device": instance["boot_volume_attachment"]["device"]["id"],
+ "name": instance["boot_volume_attachment"]["volume"]["name"]
+ },
+ "system": {
+ "arch": instance["vcpu"]["architecture"],
+ "num_cpus": instance["vcpu"]["count"],
+ "ram": instance["memory"]
+ }
+ }
+
+
+def main():
+ init()
+
+ ci_agents = {}
+
+ for instance in list_instances():
+ if CI_AGENT_RE.match(instance["name"]):
+ load_ci_agent(ci_agents, instance)
+ continue
+
+ print "Unknown instance: " + instance["name"]
+ exit(2)
+
+ inventory = {"all": {
+ "children": {
+ "ci_agents": {
+ "hosts": ci_agents
+ }
+ }
+ }}
+
+ print yaml.dump(tostr(inventory), default_flow_style=False)
+
+
+if __name__ == "__main__":
+ main()