You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by as...@apache.org on 2021/04/22 10:41:08 UTC
[airflow-ci-infra] branch main updated: User Packer to build a
pre-built AMI with everything we need (#15)
This is an automated email from the ASF dual-hosted git repository.
ash pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git
The following commit(s) were added to refs/heads/main by this push:
new 4ff4feb User Packer to build a pre-built AMI with everything we need (#15)
4ff4feb is described below
commit 4ff4feb24e5efbb1938fe4f0a9265fe73c6c2fb2
Author: Mike Hewitt <hm...@vt.edu>
AuthorDate: Thu Apr 22 06:41:02 2021 -0400
User Packer to build a pre-built AMI with everything we need (#15)
* initial packer and tf
* packer added files a scripts from Ashs repo
* add new folder structure and terraform
* updateing packer files
* added dependencies file permission and apt source repos
* bootstrap and user data
* prepare packer provisioners and set up all files to be executed
* update tinder
* terraform to create packer roles, starting to fill in packer variables
* packer roles added aws backends, terraform reformed and added iam roles as well as autoscaling cloudwatch alarm and policy
* fixed iam role and removed policy attatchments
* first run of packer_roles, terraform add gitignore for terraform
* update packer code from results of validate
* update runner max size of asg
* packer updated to run and terraform roles for packer updated
* Apply suggestions from code review
* Update for pre-commit checks
Add licenses, and remove trailing whitespace
* archieve lambda before upload
* remove terraform for ci infra
* Make the packer build produce a working image.
Summary of changes:
- Files need to be copied to a "staging" folder and then moved in place
- Use the built-in upload ability of the shell provisioner
- Have shell provisioner run scripts with sudo, rather than using sudo
10s of times in the scripts
- Don't set up tmpfs mounts in the AMI -- these have to happen at
instance boot time, not AMI creation
- Preseed the install options for iptables-persistent so that it
installs without asking questions or replacing the rules we already
placed.
- Install the runner-supervisor script from local file, not S3.
Co-authored-by: Ash Berlin-Taylor <as...@firemirror.com>
---
.gitignore | 35 +++++
.../packer/files/actions-runner-ec2-reporting.sh | 22 +++
.../packer/files/actions.runner-supervisor.service | 30 ++++
.../packer/files/actions.runner.service | 38 +++++
.../packer/files/cloudwatch-metrics-github-runners | 18 +++
github-runner-ami/packer/files/docker-compose.sh | 25 +++
.../packer/files/install-dependencies.sh | 41 +++++
github-runner-ami/packer/files/install-files.sh | 27 ++++
github-runner-ami/packer/files/mounts_setup.sh | 20 +++
github-runner-ami/packer/files/rules.v4 | 28 ++++
github-runner-ami/packer/files/runner | 18 +++
.../packer/files/runner-cleanup-workdir.sh | 35 +++++
.../packer/files}/runner-supervisor.py | 0
github-runner-ami/packer/files/runner_bootstrap.sh | 42 +++++
.../packer/files/source-list-additions.sh | 23 +++
.../packer/files/stop-runner-if-no-job.sh | 43 ++++++
github-runner-ami/packer/files/timber.key | 36 +++++
github-runner-ami/packer/files/vector.toml | 98 ++++++++++++
github-runner-ami/packer/ubuntu2004.pkr.hcl | 148 ++++++++++++++++++
.../packer/vars/variables.pkrvars.hcl | 25 +++
terraform/packer_roles/.terraform.lock.hcl | 38 +++++
terraform/packer_roles/main.tf | 171 +++++++++++++++++++++
terraform/packer_roles/output.tf | 20 +++
23 files changed, 981 insertions(+)
diff --git a/.gitignore b/.gitignore
index 1c97fe3..c3e2a8c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,37 @@
.cache
__pycache__/
+
+# Created by https://www.toptal.com/developers/gitignore/api/terraform
+# Edit at https://www.toptal.com/developers/gitignore?templates=terraform
+
+### Terraform ###
+# Local .terraform directories
+**/.terraform/*
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+
+# Crash log files
+crash.log
+
+# Ignore any .tfvars files that are generated automatically for each Terraform run. Most
+# .tfvars files are managed as part of configuration and so should be included in
+# version control.
+#
+# example.tfvars
+
+# Ignore override files as they are usually used to override resources locally and so
+# are not checked in
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json
+
+# Include override files you do wish to add to version control using negated pattern
+# !example_override.tf
+
+# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
+# example: *tfplan*
+
+# End of https://www.toptal.com/developers/gitignore/api/terraform
diff --git a/github-runner-ami/packer/files/actions-runner-ec2-reporting.sh b/github-runner-ami/packer/files/actions-runner-ec2-reporting.sh
new file mode 100644
index 0000000..d79b0c9
--- /dev/null
+++ b/github-runner-ami/packer/files/actions-runner-ec2-reporting.sh
@@ -0,0 +1,22 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+if pgrep -c Runner.Worker >/dev/null; then
+ # Only report metric when we're doing something -- no point paying to submit zeros
+ aws cloudwatch put-metric-data --metric-name jobs-running --value "$(pgrep -c Runner.Worker)" --namespace github.actions
+fi
diff --git a/github-runner-ami/packer/files/actions.runner-supervisor.service b/github-runner-ami/packer/files/actions.runner-supervisor.service
new file mode 100644
index 0000000..baffc90
--- /dev/null
+++ b/github-runner-ami/packer/files/actions.runner-supervisor.service
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[Unit]
+Description=Fetch credentials and supervise GitHub Actions Runner
+After=network.target
+Before=actions.runner.service
+
+[Service]
+Type=notify
+ExecStart=/opt/runner-supervisor/bin/python /opt/runner-supervisor/bin/runner-supervisor
+# We need to run as root to have the ability to open the netlink connector socket
+User=root
+WorkingDirectory=/home/runner/actions-runner
+Restart=always
+EnvironmentFile=/etc/environment
diff --git a/github-runner-ami/packer/files/actions.runner.service b/github-runner-ami/packer/files/actions.runner.service
new file mode 100644
index 0000000..5c2e6d0
--- /dev/null
+++ b/github-runner-ami/packer/files/actions.runner.service
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[Unit]
+Description=GitHub Actions Runner
+After=network.target actions.runner-supervisor.service
+Requires=actions.runner-supervisor.service
+BindsTo=actions.runner-supervisor.service
+
+[Service]
+ExecStartPre=!/usr/local/sbin/runner-cleanup-workdir.sh
+ExecStart=/home/runner/actions-runner/run.sh --once --startuptype service
+ExecStop=/usr/local/bin/stop-runner-if-no-job.sh $MAINPID
+EnvironmentFile=/etc/environment
+Environment=GITHUB_ACTIONS_RUNNER_CHANNEL_TIMEOUT=300
+User=runner
+WorkingDirectory=/home/runner/actions-runner
+KillMode=mixed
+KillSignal=SIGTERM
+TimeoutStopSec=5min
+Restart=on-success
+
+[Install]
+WantedBy=multi-user.target
diff --git a/github-runner-ami/packer/files/cloudwatch-metrics-github-runners b/github-runner-ami/packer/files/cloudwatch-metrics-github-runners
new file mode 100644
index 0000000..fb7f8eb
--- /dev/null
+++ b/github-runner-ami/packer/files/cloudwatch-metrics-github-runners
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+*/1 * * * * nobody /usr/local/sbin/actions-runner-ec2-reporting
diff --git a/github-runner-ami/packer/files/docker-compose.sh b/github-runner-ami/packer/files/docker-compose.sh
new file mode 100644
index 0000000..378b7e5
--- /dev/null
+++ b/github-runner-ami/packer/files/docker-compose.sh
@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -exu -o pipefail
+
+# https://github.com/actions/virtual-environments/blob/525f79f479cca77aef4e0a680548b65534c64a18/images/linux/scripts/installers/docker-compose.sh
+URL=$(curl -s https://api.github.com/repos/docker/compose/releases/latest | jq -r '.assets[].browser_download_url | select(endswith("docker-compose-Linux-x86_64"))')
+curl --fail -L "$URL" -o /usr/local/bin/docker-compose
+chmod +x /usr/local/bin/docker-compose
diff --git a/github-runner-ami/packer/files/install-dependencies.sh b/github-runner-ami/packer/files/install-dependencies.sh
new file mode 100644
index 0000000..0b20557
--- /dev/null
+++ b/github-runner-ami/packer/files/install-dependencies.sh
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+export DEBIAN_FRONTEND=noninteractive
+
+debconf-set-selections <<EOF
+iptables-persistent iptables-persistent/autosave_done boolean true
+iptables-persistent iptables-persistent/autosave_v4 boolean false
+iptables-persistent iptables-persistent/autosave_v6 boolean false
+EOF
+
+apt-get update
+apt-get install -y --no-install-recommends \
+ awscli \
+ build-essential \
+ docker.io \
+ git \
+ iptables-persistent \
+ jq \
+ parallel \
+ python3-dev \
+ python3-venv \
+ python3-wheel \
+ yarn \
+ vector
diff --git a/github-runner-ami/packer/files/install-files.sh b/github-runner-ami/packer/files/install-files.sh
new file mode 100644
index 0000000..b99e472
--- /dev/null
+++ b/github-runner-ami/packer/files/install-files.sh
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -eu -o pipefail
+
+mkdir /etc/iptables/
+
+install --owner root --mode=0644 --target-directory "/etc/systemd/system/" "/tmp/etc-systemd-system/"*
+install --owner root --mode=0755 --target-directory "/usr/local/sbin" "/tmp/usr-local-sbin/"*
+install --owner root --mode=0644 --target-directory "/etc/iptables" "/tmp/etc-iptables/"*
+install --owner root --mode=0644 --target-directory "/etc/cron.d" "/tmp/etc-cron.d/"*
+install --owner root --mode=0644 --target-directory "/etc/sudoers.d" "/tmp/etc-sudoers.d/"*
diff --git a/github-runner-ami/packer/files/mounts_setup.sh b/github-runner-ami/packer/files/mounts_setup.sh
new file mode 100644
index 0000000..f8bee50
--- /dev/null
+++ b/github-runner-ami/packer/files/mounts_setup.sh
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+sudo mount -t tmpfs -o size=10% tmpfs /tmp
+sudo mount -t tmpfs -o size=66% tmpfs /var/lib/docker
+sudo mount -t tmpfs -o tmpfs /home/runner/actions-runner/_work
diff --git a/github-runner-ami/packer/files/rules.v4 b/github-runner-ami/packer/files/rules.v4
new file mode 100644
index 0000000..6d19ac1
--- /dev/null
+++ b/github-runner-ami/packer/files/rules.v4
@@ -0,0 +1,28 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Generated by iptables-save v1.8.4 on Thu Jan 14 13:59:27 2021
+*filter
+:INPUT ACCEPT [833:75929]
+:FORWARD DROP [0:0]
+:OUTPUT ACCEPT [794:143141]
+:DOCKER-USER - [0:0]
+-A FORWARD -j DOCKER-USER
+# Dis-allow any docker container to access the metadata service
+-A DOCKER-USER -d 169.254.169.254/32 -j REJECT --reject-with icmp-port-unreachable
+-A DOCKER-USER -j RETURN
+COMMIT
diff --git a/github-runner-ami/packer/files/runner b/github-runner-ami/packer/files/runner
new file mode 100644
index 0000000..1d79760
--- /dev/null
+++ b/github-runner-ami/packer/files/runner
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+runner ALL=(ALL) NOPASSWD:/usr/sbin/swapoff -a, /usr/bin/rm -f /swapfile, /usr/bin/apt clean
diff --git a/github-runner-ami/packer/files/runner-cleanup-workdir.sh b/github-runner-ami/packer/files/runner-cleanup-workdir.sh
new file mode 100644
index 0000000..7a1e8e8
--- /dev/null
+++ b/github-runner-ami/packer/files/runner-cleanup-workdir.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+echo "Left-over containers:"
+docker ps -a
+docker ps -qa | xargs --verbose --no-run-if-empty docker rm -fv
+
+if [[ -d ~runner/actions-runner/_work/airflow/airflow ]]; then
+ cd ~runner/actions-runner/_work/airflow/airflow
+
+ chown --changes -R runner: .
+ if [[ -e .git ]]; then
+ sudo -u runner bash -c "
+ git reset --hard && \
+ git submodule deinit --all -f && \
+ git submodule foreach git clean -fxd && \
+ git clean -fxd \
+ "
+ fi
+fi
diff --git a/scripts/runner-supervisor.py b/github-runner-ami/packer/files/runner-supervisor.py
similarity index 100%
rename from scripts/runner-supervisor.py
rename to github-runner-ami/packer/files/runner-supervisor.py
diff --git a/github-runner-ami/packer/files/runner_bootstrap.sh b/github-runner-ami/packer/files/runner_bootstrap.sh
new file mode 100644
index 0000000..0b258a7
--- /dev/null
+++ b/github-runner-ami/packer/files/runner_bootstrap.sh
@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -exu -o pipefail
+
+# Validate params
+: "${RUNNER_VERSION?}"
+
+# Set an env var (that is visible in runners) that will let us know we are on a self-hosted runner
+echo 'AIRFLOW_SELF_HOSTED_RUNNER="[\"self-hosted\"]"' >> /etc/environment
+
+useradd --create-home runner -G docker
+
+install --owner runner --directory ~runner/actions-runner
+
+cd ~runner/actions-runner
+curl -L "https://github.com/ashb/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz" | tar -zx
+
+python3 -mvenv /opt/runner-supervisor
+/opt/runner-supervisor/bin/pip install -U pip python-dynamodb-lock-whatnick==0.9.3 click==7.1.2 psutil 'tenacity~=6.0'
+
+install --owner root --mode 0755 /tmp/runner-supervisor /opt/runner-supervisor/bin/runner-supervisor
+
+systemctl enable iptables.service
+systemctl enable vector.service
+systemctl enable actions.runner.service
diff --git a/github-runner-ami/packer/files/source-list-additions.sh b/github-runner-ami/packer/files/source-list-additions.sh
new file mode 100644
index 0000000..6f8b0af
--- /dev/null
+++ b/github-runner-ami/packer/files/source-list-additions.sh
@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+apt-key adv --recv-key 0x1646B01B86E50310 "1646B01B86E50310"
+echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list
+apt-key add /tmp/timber.key
+echo "deb https://repositories.timber.io/public/vector/deb/ubuntu focal main" > /etc/apt/sources.list.d/timber.list
diff --git a/github-runner-ami/packer/files/stop-runner-if-no-job.sh b/github-runner-ami/packer/files/stop-runner-if-no-job.sh
new file mode 100644
index 0000000..cfa2a7f
--- /dev/null
+++ b/github-runner-ami/packer/files/stop-runner-if-no-job.sh
@@ -0,0 +1,43 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -u
+
+MAINPID="${MAINPID:-${1:-}}"
+
+if [[ -z "$MAINPID" ]]; then
+ echo "No MAINPID, assuming it already crashed!"
+ exit 0
+fi
+
+if pgrep --ns $MAINPID -a Runner.Worker > /dev/null; then
+ echo "Waiting for current job to finish"
+ while pgrep --ns $MAINPID -a Runner.Worker; do
+ # Job running -- just wait for it to exit
+ sleep 10
+ done
+fi
+
+# Request shutdown if it's still alive -- because we are in "stop" state it should not restart
+if pkill --ns $MAINPID Runner.Listener; then
+ # Wait for it to shut down
+ echo "Waiting for main Runner.Listener process to stop"
+ while pgrep --ns $MAINPID -a Runner.Listener; do
+ sleep 5
+ done
+fi
diff --git a/github-runner-ami/packer/files/timber.key b/github-runner-ami/packer/files/timber.key
new file mode 100644
index 0000000..f8b8df7
--- /dev/null
+++ b/github-runner-ami/packer/files/timber.key
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+Version: GnuPG v2
+
+mQENBF9gFZ0BCADETtIHM8y5ehMoyNiZcriK+tHXyKnbZCKtMCKcC4ll94/6pekQ
+jKIPWg8OXojkCtwua/TsddtQmOhUxAUtv6K0jO8r6sJ8rezMhuNH8J8rMqWgzv9d
+2+U7Z7GFgcP0OeD+KigtnR8uyp50suBmEDC8YytmmbESmG261Y38vZME0VvQ+CMy
+Yi/FvKXBXugaiCtaz0a5jVE86qSZbKbuaTHGiLn05xjTqc4FfyP4fi4oT2r6GGyL
+Bn5ob84OjXLQwfbZIIrNFR10BvL2SRLL0kKKVlMBBADodtkdwaTt0pGuyEJ+gVBz
+629PZBtSrwVRU399jGSfsxoiLca9//c7OJzHABEBAAG0OkNsb3Vkc21pdGggUGFj
+a2FnZSAodGltYmVyL3ZlY3RvcikgPHN1cHBvcnRAY2xvdWRzbWl0aC5pbz6JATcE
+EwEIACEFAl9gFZ0CGy8FCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQNUPbLQor
+xLhf6gf8DyfIpKjvEeW/O8lRUTpkiPKezJbb+udZboCXJKDD02Q9PE3hfEfQRr5X
+muytL7YMPvzqBVuP3xV5CN3zvtiQQbZiDhstImVyd+t24pQTkjzkvy+A2yvUuIkE
+RWxuey41f5FNj/7wdfJnHoU9uJ/lvsb7DLXw7FBMZFNBR6LED/d+b61zMzVvmFZA
+gsrCGwr/jfySwnpShmKdJaMTHQx0qt2RfXwNm2V6i900tAuMUWnmUIz5/9vENPKm
+0+31I43a/QgmIrKEePhwn2jfA1oRlYzdv+PbblSTfjTStem+GqQkj9bZsAuqVH8g
+3vq0NvX0k2CLi/W9mTiSdHXFChI15A==
+=k36w
+-----END PGP PUBLIC KEY BLOCK-----
diff --git a/github-runner-ami/packer/files/vector.toml b/github-runner-ami/packer/files/vector.toml
new file mode 100644
index 0000000..05db414
--- /dev/null
+++ b/github-runner-ami/packer/files/vector.toml
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+data_dir = "/var/lib/vector"
+
+[api]
+ enabled = true
+
+# Input data. Change me to a valid input source.
+[sources.logs]
+ type = "journald"
+ include_units = ["actions.runner.service", "actions.runner-supervisor.service"]
+
+[transforms.without_systemd_fields]
+ type = "remove_fields"
+ inputs = ["logs"]
+ fields = ["_CAP_EFFECTIVE", "_SYSTEMD_SLICE", "_SYSTEMD_CGROUP",
+ "_SYSTEMD_INVOCATION_ID", "_SELINUX_CONTEXT", "_COMM", "_BOOT_ID",
+ "_MACHINE_ID", "_STREAM_ID", "_PID", "_GID", "_UID","_TRANSPORT",
+ "__MONOTONIC_TIMESTAMP", "SYSLOG_IDENTIFIER", "PRIORITY",
+ "source_type"]
+
+[sources.runner-logs]
+ type = "file"
+ include = ["/home/runner/actions-runner/_diag/*.log"]
+
+ [sources.runner-logs.multiline]
+ start_pattern = '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
+ mode = "halt_before"
+ condition_pattern = '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
+ timeout_ms = 250
+
+[transforms.grok-runner-logs]
+ type = "remap"
+ inputs=["runner-logs"]
+ source = '''
+ structured, err = parse_grok(.message, "(?m)\\[%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{NOTSPACE:logger}\\] %{GREEDYDATA:message}")
+
+ if err != null {
+ .err = err
+ } else {
+ . = merge(., structured)
+ }
+ '''
+[transforms.filter-runner-logs]
+ type = "filter"
+ inputs = ['grok-runner-logs']
+ condition.type = "remap"
+ condition.source = '''
+ if .logger == "JobServerQueue" {
+ !match!(.message, r'Try to append \d+ batches web console lines for record')
+ } else if .logger == "HostContext" {
+ !starts_with!(.message, "Well known directory")
+ } else {
+ true
+ }
+ '''
+
+[sources.job-logs]
+ type = "file"
+ include = ["/home/runner/actions-runner/_diag/pages/*.log"]
+
+[transforms.grok-job-logs]
+ type = "remap"
+ inputs = ["job-logs"]
+ source = '''
+ structured, err = parse_grok(.message, "%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:message}")
+
+ if err == null {
+ . = merge(., structured)
+ .type = "job-output"
+ }
+ '''
+
+# Output data
+[sinks.cloudwatch]
+inputs = ["without_systemd_fields", "filter-runner-logs", "grok-job-logs"]
+type = "aws_cloudwatch_logs"
+encoding = "json"
+create_missing_group = false
+create_missing_stream = true
+group_name = "GitHubRunners"
+stream_name = "{{ host }}"
+region = "eu-central-1"
diff --git a/github-runner-ami/packer/ubuntu2004.pkr.hcl b/github-runner-ami/packer/ubuntu2004.pkr.hcl
new file mode 100644
index 0000000..93f8799
--- /dev/null
+++ b/github-runner-ami/packer/ubuntu2004.pkr.hcl
@@ -0,0 +1,148 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+variable "vpc_id" {
+ type = string
+}
+variable "ami_name" {
+ type = string
+}
+variable "aws_region" {
+ type = string
+}
+variable "subnet_id" {
+ type = string
+}
+variable "packer_role_arn" {
+ type = string
+}
+variable "runner_version" {
+ type = string
+}
+variable "kms_key_arn" {
+ type = string
+}
+variable "session_manager_instance_profile_name" {
+ type = string
+}
+
+source "amazon-ebs" "runner_builder" {
+ assume_role {
+ role_arn = var.packer_role_arn
+ session_name = var.runner_version
+ }
+ #access_key = ""
+ #secret_key = ""
+ region = var.aws_region
+ ami_name = "${var.ami_name}-${var.runner_version}"
+ ami_regions = [var.aws_region]
+ tag {
+ key = "ami"
+ value = "github-runner-ami"
+ }
+ encrypt_boot = true
+ kms_key_id = var.kms_key_arn
+ instance_type = "t3.micro"
+ communicator = "ssh"
+ ssh_username = "ubuntu"
+ ssh_interface = "session_manager"
+ iam_instance_profile = var.session_manager_instance_profile_name
+ subnet_id = var.subnet_id
+ vpc_id = var.vpc_id
+ source_ami_filter {
+ filters = {
+ virtualization-type = "hvm"
+ name = "ubuntu/images/*buntu-focal-20.04-amd64-server-*"
+ root-device-type = "ebs"
+ }
+ owners = ["099720109477"]
+ most_recent = true
+ }
+}
+
+build {
+ sources = [
+ "source.amazon-ebs.runner_builder"
+ ]
+
+ provisioner "shell" {
+ inline = [
+ "echo Connected via SSM at '${build.User}@${build.Host}:${build.Port}'"
+ ]
+ }
+
+ # Since we connect as a non-root user, we have to "stage" the files to a writable folder, which we then move
+ # in to place with the approriate permissions via install-files.sh provisioner step
+ provisioner "shell" {
+ inline = [
+ "mkdir -p /tmp/etc-systemd-system /tmp/usr-local-sbin /tmp/etc-sudoers.d /tmp/etc-iptables /tmp/etc-cron.d"
+ ]
+ }
+ provisioner "file" {
+ destination = "/tmp/etc-systemd-system/actions.runner.service"
+ source = "./files/actions.runner.service"
+ }
+ provisioner "file" {
+ destination = "/tmp/etc-systemd-system/actions.runner-supervisor.service"
+ source = "./files/actions.runner-supervisor.service"
+ }
+ provisioner "file" {
+ destination = "/tmp/usr-local-sbin/runner-cleanup-workdir.sh"
+ source = "./files/runner-cleanup-workdir.sh"
+ }
+ provisioner "file" {
+ destination = "/tmp/usr-local-sbin/stop-runner-if-no-job.sh"
+ source = "./files/stop-runner-if-no-job.sh"
+ }
+ provisioner "file" {
+ destination = "/tmp/etc-sudoers.d/runner"
+ source = "./files/runner"
+ }
+ provisioner "file" {
+ destination = "/tmp/etc-iptables/rules.v4"
+ source = "./files/rules.v4"
+ }
+ provisioner "file" {
+ destination = "/tmp/usr-local-sbin/actions-runner-ec2-reporting.sh"
+ source = "./files/actions-runner-ec2-reporting.sh"
+ }
+ provisioner "file" {
+ destination = "/tmp/etc-cron.d/cloudwatch-metrics-github-runners"
+ source = "./files/cloudwatch-metrics-github-runners"
+ }
+ provisioner "file" {
+ destination = "/tmp/timber.key"
+ source = "./files/timber.key"
+ }
+ provisioner "file" {
+ destination = "/tmp/runner-supervisor"
+ source = "./files/runner-supervisor.py"
+ }
+ provisioner "shell" {
+ scripts = [
+ "./files/install-files.sh",
+ "./files/source-list-additions.sh",
+ "./files/install-dependencies.sh",
+ "./files/docker-compose.sh",
+ "./files/runner_bootstrap.sh",
+ ]
+ execute_command = "chmod +x '{{ .Path }}'; sudo sh -c '{{ .Vars }} {{ .Path }}'"
+ environment_vars = [
+ "RUNNER_VERSION=${var.runner_version}",
+ ]
+ }
+}
diff --git a/github-runner-ami/packer/vars/variables.pkrvars.hcl b/github-runner-ami/packer/vars/variables.pkrvars.hcl
new file mode 100644
index 0000000..2680fdd
--- /dev/null
+++ b/github-runner-ami/packer/vars/variables.pkrvars.hcl
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+vpc_id = "vpc-d73487bd"
+ami_name = "airflow-runner-ami"
+aws_region = "eu-central-1"
+subnet_id = "subnet-72ed3c0e"
+packer_role_arn = "arn:aws:iam::827901512104:role/packer-role"
+runner_version = "2.277.1-airflow3"
+kms_key_arn = "arn:aws:kms:eu-central-1:827901512104:key/48a58710-7ac6-4f88-995f-758a6a450faa"
+session_manager_instance_profile_name = "packer_ssm_instance_profile"
diff --git a/terraform/packer_roles/.terraform.lock.hcl b/terraform/packer_roles/.terraform.lock.hcl
new file mode 100644
index 0000000..51c16e0
--- /dev/null
+++ b/terraform/packer_roles/.terraform.lock.hcl
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This file is maintained automatically by "terraform init".
+# Manual edits may be lost in future updates.
+
+provider "registry.terraform.io/hashicorp/aws" {
+ version = "3.34.0"
+ constraints = "~> 3.0"
+ hashes = [
+ "h1:2xGmnG7UF0iScMGVWBazSYk1sRatXeZYgCGjdGm4A+w=",
+ "zh:003272229bd19bb63d6e77bc3d684268c417a151dfaee01c40b40e21cdd8bb0f",
+ "zh:103cacc1f3d97dfb7e9dd1e1905b075f92d9bd8aed434f811e8111788b648a57",
+ "zh:63a43c6e5fb2e5ad59ea068bede5c6bb54358affd32163d72785473a15440427",
+ "zh:6648af39a318c85eb336e2fb3ec1a01c5ffe8d75cc51686c37e892dd6f6a8974",
+ "zh:71ac8f6d5d61e5dee90099fd4fc1bb5bcd8ccb674eb6e7cd58d20757f7cecd12",
+ "zh:73baae4aa5bc0af12917e3bb17e1086050d25cdf7ba604f7fc422653c99f884c",
+ "zh:7d920ac05c45e77c59c49e0dd0cb010d64202c5a2fdfde6d9efe3dc61e396c97",
+ "zh:8a495e49f8fcbe276a74911f9ca48381533686ff71a9d4f7027bb9109769b639",
+ "zh:8ab9769581dfc1675c645e33e7ab8fea6ad1acc9e232eeda823070447e5ecaf1",
+ "zh:a170ecc560d49c251f4bebb6d6a82ff3637ae16a0f779a53489d4a64ddd1ee6a",
+ "zh:d9178201057b62666691ec206d1fbe09965bcfea532085b4e31f46073bf5898f",
+ ]
+}
diff --git a/terraform/packer_roles/main.tf b/terraform/packer_roles/main.tf
new file mode 100644
index 0000000..d275ad4
--- /dev/null
+++ b/terraform/packer_roles/main.tf
@@ -0,0 +1,171 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+terraform {
+ required_providers {
+ aws = {
+ source = "hashicorp/aws"
+ version = "~> 3.0"
+ }
+ }
+ backend "s3" {
+ bucket = "airflow-ci-tfstate"
+ key = "tf-state/packer-roles"
+ region = "us-east-2"
+ }
+}
+
+# Configure the AWS Provider
+provider "aws" {
+ region = "us-east-2"
+}
+
+### PACKER NEED IAM ROLE ###
+resource "aws_iam_role_policy" "packer_policy" {
+ name = "packer-base-policy"
+ role = aws_iam_role.packer_role.id
+
+ policy = <<-EOF
+ {
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Sid": "04092021",
+ "Effect": "Allow",
+ "Action": [
+ "ec2:AttachVolume",
+ "ec2:AuthorizeSecurityGroupIngress",
+ "ec2:CopyImage",
+ "ec2:CreateImage",
+ "ec2:CreateKeypair",
+ "ec2:CreateSecurityGroup",
+ "ec2:CreateSnapshot",
+ "ec2:CreateTags",
+ "ec2:CreateVolume",
+ "ec2:DeleteKeyPair",
+ "ec2:DeleteSecurityGroup",
+ "ec2:DeleteSnapshot",
+ "ec2:DeleteVolume",
+ "ec2:DeregisterImage",
+ "ec2:DescribeImageAttribute",
+ "ec2:DescribeImages",
+ "ec2:DescribeInstances",
+ "ec2:DescribeInstanceStatus",
+ "ec2:DescribeRegions",
+ "ec2:DescribeSecurityGroups",
+ "ec2:DescribeSnapshots",
+ "ec2:DescribeSubnets",
+ "ec2:DescribeTags",
+ "ec2:DescribeVolumes",
+ "ec2:DetachVolume",
+ "ec2:GetPasswordData",
+ "ec2:ModifyImageAttribute",
+ "ec2:ModifyInstanceAttribute",
+ "ec2:ModifySnapshotAttribute",
+ "ec2:RegisterImage",
+ "ec2:RunInstances",
+ "ec2:StopInstances",
+ "ec2:TerminateInstances"
+ ],
+ "Resource": "*"
+ }
+ ]
+ }
+ EOF
+}
+
+resource "aws_iam_role" "packer_role" {
+ name = "packer-role"
+
+ assume_role_policy = <<EOF
+{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Action": "sts:AssumeRole",
+ "Principal": {
+ "Service": "ec2.amazonaws.com"
+ },
+ "Effect": "Allow",
+ "Sid": ""
+ }
+ ]
+}
+EOF
+
+ tags = {
+ tag-key = "tag-value"
+ }
+}
+
+### SSM ###
+
+resource "aws_iam_role" "packer_ssm_role" {
+ name = "packer-ssm-role"
+
+ assume_role_policy = <<EOF
+{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Action": "sts:AssumeRole",
+ "Principal": {
+ "Service": "ec2.amazonaws.com"
+ },
+ "Effect": "Allow",
+ "Sid": ""
+ }
+ ]
+}
+EOF
+
+ tags = {
+ tag-key = "tag-value"
+ }
+}
+
+resource "aws_iam_role_policy" "packer_ssm_policy" {
+ name = "packer-ssm-policy"
+ role = aws_iam_role.packer_ssm_role.id
+
+ policy = <<-EOF
+ {
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Sid": "04092021",
+ "Effect": "Allow",
+ "Action": [
+ "ssm:StartSession"
+ ],
+
+ "Resource": "arn:aws:ec2:*:*:instance/*"
+ }
+ ]
+ }
+ EOF
+}
+
+resource "aws_iam_role_policy_attachment" "attach_packer_ssm_policy" {
+ role = aws_iam_role.packer_ssm_role.name
+ policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
+}
+
+resource "aws_iam_instance_profile" "packer_ssm_profile" {
+ name = "packer_ssm_instance_profile"
+ role = aws_iam_role.packer_ssm_role.name
+}
diff --git a/terraform/packer_roles/output.tf b/terraform/packer_roles/output.tf
new file mode 100644
index 0000000..6b265ed
--- /dev/null
+++ b/terraform/packer_roles/output.tf
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+output "packer_role" {
+ value = aws_iam_role.packer_role.arn
+}