You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by sh...@apache.org on 2026/09/17 10:01:15 UTC
(airflow-ci-infra) 34/34: Replace static node groups with Karpenter and ARC runner scale sets
This is an automated email from the ASF dual-hosted git repository.
shahar1 pushed a commit to branch gha-arc-rebase
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git
commit aec106759990dbfe529c1c8b4fe4cfb1a0cfd3df
Author: Shahar Epstein <60...@users.noreply.github.com>
AuthorDate: Thu Sep 17 13:00:36 2026 +0300
Replace static node groups with Karpenter and ARC runner scale sets
The previous design (cluster-autoscaler, six managed node groups, legacy
ARC with cert-manager) could not scale from zero without ASG tag
workarounds and had no spot-price-aware instance selection. The CodeBuild
experiment measured ~$104 per full Tests (AMD) run against a ~$15k/yr CI
budget. Karpenter with EKS Pod Identity and ARC's supported scale-set mode
brings the same run to ~$12-13 with one NodePool per architecture and no
cert-manager or EBS CSI dependency.
The design rationale lives in docs/eks-runner-architecture.md and ADR 0001;
the runner image now matches the host contract airflow's CI graph expects.
---
Airflow_CI.png | Bin 80422 -> 0 bytes
CONTEXT.md | 47 ++++
README.rst | 122 +++++------
...ks-standard-mode-with-self-managed-karpenter.md | 40 ++++
docs/eks-runner-architecture.md | 244 +++++++++++++++++++++
helm/{ci-helmfile.yaml => ci-helmfile.yaml.gotmpl} | 15 +-
.../templates/namespaces/{infra.yaml => arc.yaml} | 9 +-
helm/cluster-config/templates/namespaces/ci.yaml | 26 ---
helm/infra-helmfile.yaml | 42 ++--
.../Chart.yaml} | 9 +-
.../templates/ec2nodeclass.yaml | 71 ++++++
.../templates/nodepool-amd-shared.yaml | 65 ++++++
.../templates/nodepool-amd.yaml | 63 ++++++
.../templates/nodepool-arm-shared.yaml | 65 ++++++
.../templates/nodepool-arm.yaml | 63 ++++++
.../values.yaml} | 30 +--
helm/values/cert-manager.yaml | 20 --
helm/values/gha-runner-scale-set-controller.yaml | 5 +-
.../gha-runner-scale-sets/common.yaml.gotmpl | 137 ++++++++++--
helm/values/gha-runner-scale-sets/runners.yaml | 97 +++++---
.../{cluster-autoscaler.yaml => karpenter.yaml} | 28 +--
runner/Dockerfile | 88 +++++++-
terraform/README.rst | 26 ++-
terraform/eks/.terraform.lock.hcl | 145 ++++++------
terraform/eks/asg_tags.tf | 67 ------
terraform/eks/eks.tf | 233 ++++----------------
terraform/eks/iam.tf | 134 -----------
terraform/eks/{vpc.tf => karpenter.tf} | 38 ++--
terraform/eks/{vpc.tf => outputs.tf} | 51 ++---
terraform/eks/provider.tf | 13 +-
terraform/eks/runner_pod_identity.tf | 113 ++++++++++
terraform/eks/state.tf | 16 +-
terraform/eks/variables.tf | 48 +---
terraform/eks/version.tf | 5 +-
terraform/eks/vpc.tf | 22 +-
35 files changed, 1376 insertions(+), 821 deletions(-)
diff --git a/Airflow_CI.png b/Airflow_CI.png
deleted file mode 100644
index 82e5043..0000000
Binary files a/Airflow_CI.png and /dev/null differ
diff --git a/CONTEXT.md b/CONTEXT.md
new file mode 100644
index 0000000..901412a
--- /dev/null
+++ b/CONTEXT.md
@@ -0,0 +1,47 @@
+# Airflow CI Infrastructure
+
+Self-hosted GitHub Actions runner infrastructure (EKS + ARC + Karpenter) that offloads
+selected Apache Airflow CI workloads from GitHub-hosted runners to AWS.
+
+## Language
+
+### Workload categories
+
+**Trusted run**:
+A workflow run whose code ref is not attacker-controllable: `schedule`, `push`, or
+`workflow_dispatch` events on `apache/airflow` itself. The only category eligible for
+self-hosted runners in phase 1.
+_Avoid_: using "trusted" for committer-authored PRs — those are Committer runs.
+
+**Canary run**:
+A full-matrix CI run of main, triggered by cron (`schedule`) or by a push to main.
+"Scheduled canary" and "push canary" distinguish the trigger when it matters.
+
+**Committer run**:
+A CI run of a PR authored by someone with merge access (regardless of whether it comes
+from a fork or a branch). Not a Trusted run — the PR code is still unreviewed; serving
+these from self-hosted runners is a separate, unresolved decision.
+_Avoid_: trusted PR, member run.
+
+**Fork PR run**:
+A CI run of a PR from a contributor without merge access. Never eligible for
+self-hosted runners.
+
+### Infrastructure
+
+**Runner scale set**:
+An ARC unit of runner capacity targeted by a single `runs-on` label (e.g.
+`airflow-runner-amd-xl`). One helm release per scale set.
+_Avoid_: runner deployment (legacy ARC term), runner pool.
+
+**Runner class**:
+The size tier of a scale set, expressed as pod resource requests (CPU/memory), not as a
+node type. Karpenter chooses the actual instance.
+
+**System node group**:
+The small always-on EKS managed node group hosting the controllers (ARC, Karpenter,
+listeners). Everything else runs on Karpenter-provisioned nodes.
+
+**Stash**:
+The S3-backed mechanism for passing artifacts between jobs of a run (drop-in for the
+`apache/infrastructure-actions/stash` actions).
diff --git a/README.rst b/README.rst
index 729c657..f3c1e43 100644
--- a/README.rst
+++ b/README.rst
@@ -53,7 +53,7 @@ you need to follow the steps below:
3. Update kubeconfig to access the EKS cluster::
- AWS_PROFILE=airflow aws eks --region us-east-2 update-kubeconfig --name airflow --alias airflow
+ AWS_PROFILE=airflow aws eks --region us-east-2 update-kubeconfig --name airflow-ci --alias airflow-ci
This command will update the kubeconfig file located at `~/.kube/config` to include the EKS cluster.
You can verify that the cluster is added by running `kubectl get nodes`.
@@ -65,82 +65,82 @@ you need to follow the steps below:
Airflow CI infrastructure architecture
--------------------------------------
-.. image:: Airflow_CI.png
- :width: 100%
- :align: center
- :alt: Airflow CI infrastructure architecture
-Manage the infrastructure resources
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The design is documented in `docs/eks-runner-architecture.md <docs/eks-runner-architecture.md>`_.
+In short: an EKS cluster (standard mode) with a small always-on system node group runs
+`Actions Runner Controller <https://github.com/actions/actions-runner-controller>`_ (runner
+scale sets mode) and `Karpenter <https://karpenter.sh>`_; runner pods are ephemeral
+docker-in-docker pods, one job per pod, on spot nodes that Karpenter provisions on demand
+and reclaims when idle.
-We use Terraform to manage the EKS cluster and other AWS resources, and Helm (with helmfile) to manage the
-Kubernetes resources inside the cluster.
+Deploying from scratch
+~~~~~~~~~~~~~~~~~~~~~~
-For Terraform, you can find the detailed documentation here: `Terraform Documentation <terraform/README.rst>`_
+We use Terraform for the AWS resources and Helm (with helmfile) for everything inside the
+cluster. The order matters on a fresh account:
-And for Helm, you mainly need two commands:
+1. **Terraform** — see `Terraform Documentation <terraform/README.rst>`_::
-.. code-block:: bash
+ cd terraform/eks
+ terraform init
+ terraform plan -out=terraform.tfplan
+ terraform apply terraform.tfplan
- # diff the infra helmfile to compare the current state with the desired state
- helmfile -f helm/infra-helmfile.yaml diff
+2. **kubeconfig**::
- # apply the helmfiles to deploy the desired state
- helmfile -f helm/infra-helmfile.yaml apply
+ aws eks --region us-east-2 update-kubeconfig --name airflow-ci --alias airflow-ci
- # same for ci helmfile
- helmfile -f helm/ci-helmfile.yaml diff
- helmfile -f helm/ci-helmfile.yaml apply
+3. **Infra helmfile** (namespaces, Karpenter, NodePools)::
-For Docker, we build a custom image for the runners, based on the official `Dockerfiles`_ provided by Github, but
-with the `Airflow runner release`_ managed by `Ash`_ instead of the official one, and with python installed on the image
-(check `setup-python issue`_ for more details).
+ helmfile -f helm/infra-helmfile.yaml diff
+ helmfile -f helm/infra-helmfile.yaml apply
-To build docker image, you can run the following command:
+4. **GitHub App secret** — create a GitHub App (for repository-level runners:
+ ``Administration: Read & write`` + ``Metadata: Read-only``; for organization-level
+ runners: ``Self-hosted runners: Read & write``), install it on the target
+ repo/org, then::
-.. code-block:: bash
+ kubectl create secret generic github-config --namespace arc-runners \
+ --from-literal=github_app_id=<app-id> \
+ --from-literal=github_app_installation_id=<installation-id> \
+ --from-file=github_app_private_key=<path-to-private-key.pem>
- export TARGETPLATFORM=linux/arm64
- export AIRFLOW_RUNNER_VERSION=2.304.0-airflow8
- docker build --platform $TARGETPLATFORM \
- --build-arg AIRFLOW_RUNNER_VERSION=$AIRFLOW_RUNNER_VERSION \
- -t public.ecr.aws/u9s5q9f7/airflow-gha-runner:$AIRFLOW_RUNNER_VERSION \
- runner --push
+5. **CI helmfile** (ARC controller + runner scale sets)::
-.. _Dockerfiles: https://github.com/actions/actions-runner-controller/tree/master/runner
+ helmfile -f helm/ci-helmfile.yaml.gotmpl diff
+ helmfile -f helm/ci-helmfile.yaml.gotmpl apply
-.. _Airflow runner release: https://github.com/ashb/runner/tree/v2.304.0-airflow8
+The runner image extends the official ``ghcr.io/actions/actions-runner`` image with the
+host tools airflow's CI expects (sudo, docker CLI + compose, python3, ...). Images are
+**date-tagged, never ``:latest``** — the tag referenced in
+``helm/values/gha-runner-scale-sets/runners.yaml`` must always be an explicit, reviewed
+pin. Build and push::
-.. _Ash: https://github.com/ashb
-
-.. _setup-python issue: https://github.com/actions/setup-python/issues/705
+ TAG=$(date +%Y%m%d)
+ docker build --platform linux/amd64 \
+ -t ghcr.io/apache/airflow-ci-infra/actions-runner:${TAG} runner
+ docker push ghcr.io/apache/airflow-ci-infra/actions-runner:${TAG}
+ # then update the image tag in helm/values/gha-runner-scale-sets/runners.yaml
Some important notes
~~~~~~~~~~~~~~~~~~~~
-There are some configuration parameters that you need to be aware of:
-
-- EKS node group instance type:
- In our current architecture, we use different types of runners, and each set of runners is configured via a
- RunnerDeployment. Instead of setting container resources, we define the node selectors to schedule the
- runner pod on a specific node group, and we add an anti-affinity to ensure the pod uses all the node resources
- without sharing them with other runners. In this case, we need to carefully choose the instance types because they
- represent the runners' resources.
-
-- ARC sync period:
- The ARC sync period is the duration in which the controller reconciles the desired runners count. Increasing
- this period could lead to a delay in the runners count update (more time in the queue when needing a scale-up
- and a waste of resources when needing a scale-down). Decreasing it could lead to a high load on the Github API.
-
-- Autoscaler conf:
- Similar to the ARC sync period, the autoscaler `scan-interval` is the duration in which the autoscaler checks the
- cluster resources and scales up or down the cluster. Additionally, two other important parameters are
- `scale-down-unneeded-time` and `scale-down-delay-after-add`, which control the delay before taking the scale-down
- decision when the node is underutilized and the delay of checking the scaling-down necessity after a scale-up
- decision.
-
-- HorizontalRunnerAutoscaler scaleDownDelaySecondsAfterScaleOut:
- This parameter represents the delay before the HorizontalRunnerAutoscaler (HRA) takes the scale-down decision after
- adding a new runner. Currently, we use a small delay (30 seconds) because the K8S autoscaler waits for 5 minutes
- before deleting the node. So, if we need a runner, the HRA will add a new one, and it will be scheduled in less
- than 20 seconds.
+- Runner sizing is done with pod **resource requests** (see
+ ``helm/values/gha-runner-scale-sets/runners.yaml``), not node selectors/anti-affinity:
+ Karpenter picks the cheapest spot instance that fits the request. The current single
+ ``airflow-runner-amd-xl`` class requests a whole 32-vCPU node per job.
+- Scale-up path: job queued → ARC listener requests a runner pod → pod Pending →
+ Karpenter launches a node (~60-90s). Scale-down: pod finishes → node empty →
+ Karpenter consolidates it after 60s (``WhenEmpty`` — running jobs are never evicted).
+- The dockerd inside each runner is configured (dind sidecar args in
+ ``helm/values/gha-runner-scale-sets/common.yaml.gotmpl``) with a registry mirror,
+ widened address pools and nofile ulimits — these fix real failures observed with
+ airflow's CI and should not be removed casually.
+- Workflows target the runners with ``runs-on: <scale-set-name>``, e.g.
+ ``runs-on: airflow-runner-amd-xl``.
+- **Spot policy (phase 1): spot-first for everything, deliberately.** A spot reclaim
+ kills the job mid-run — the job shows red with a "runner lost communication"
+ annotation; just re-run the failed jobs. Tripwire: if interruptions exceed ~2/week
+ (or kill the same canary twice in a row), pin the heavy runner class to on-demand
+ (``karpenter.sh/capacity-type`` requirement in the NodePool) once size classes are
+ split.
diff --git a/docs/adr/0001-eks-standard-mode-with-self-managed-karpenter.md b/docs/adr/0001-eks-standard-mode-with-self-managed-karpenter.md
new file mode 100644
index 0000000..28f7b23
--- /dev/null
+++ b/docs/adr/0001-eks-standard-mode-with-self-managed-karpenter.md
@@ -0,0 +1,40 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# EKS standard mode with self-managed Karpenter, not EKS Auto Mode
+
+Status: accepted (2026-07-25)
+
+The CI runner cluster runs EKS in standard mode with Karpenter installed and
+managed by us, even though EKS Auto Mode is AWS's recommended default and would
+remove Karpenter, AMI, and addon management entirely. Auto Mode nodes are
+locked-down Bottlerocket appliances — SELinux enforcing, immutable read-only
+root, no SSH/SSM, no node-level customization — and our runner pods are
+privileged docker-in-docker workloads that run KinD (Kubernetes *inside* that
+dind) and require node sysctls (`fs.inotify.*`, `nf_conntrack`, `fs.nr_open`);
+this is precisely the workload class that fights a locked host, and it is
+battle-tested on plain AL2023 with our own user data. Auto Mode also adds a
+per-instance management fee (~10–12% of the instance price) to a
+compute-dominated, budget-capped bill.
+
+Revisit if the workload ever stops needing privileged dind (e.g. jobs move to
+rootless/kaniko-style builds and KinD is replaced), or if Auto Mode gains
+supported node-level sysctl/privileged-workload customization at a fee that
+undercuts our measured operational cost of self-managing Karpenter
+(~a few hours per quarter).
diff --git a/docs/eks-runner-architecture.md b/docs/eks-runner-architecture.md
new file mode 100644
index 0000000..c9a71d8
--- /dev/null
+++ b/docs/eks-runner-architecture.md
@@ -0,0 +1,244 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# EKS-based GitHub Actions runners for Airflow CI — architecture
+
+*Status: proposal (2026-07). Supersedes the 2023/2024 design on this branch (PR #59).
+All versions and prices below were checked against official sources on 2026-07-25.*
+
+## Why this exists
+
+Airflow's canary/scheduled CI competes for the shared ASF GitHub Actions concurrency
+pool. An AWS CodeBuild-managed-runner experiment (June–July 2026) proved the mechanics
+work but the economics don't: a full "Tests (AMD)" run costs **≈$104** on CodeBuild
+(1,300 build-minutes on 36 vCPU machines at $0.0798/min), ≈$135k+/yr for scheduled
+runs alone — far beyond the ~$15k/yr CI budget envelope. Measured per-job utilization
+shows a **7.3× oversizing waste** (median job peaks at ~3 vCPU / 2.3 GiB on a
+36 vCPU / 72 GiB box) on top of CodeBuild's ~6.6× per-vCPU price premium over EC2 spot.
+EKS + spot brings the same run to **≈$7–13**.
+
+## Architecture at a glance
+
+```mermaid
+flowchart LR
+ subgraph github["GitHub"]
+ actions["GitHub Actions service"]
+ workflow["apache/airflow workflows<br/>runs-on: airflow-runner-amd-xl"]
+ workflow -->|"queue job"| actions
+ end
+
+ subgraph aws["AWS account · us-east-2"]
+ subgraph vpc["airflow-ci-vpc — public subnets, no NAT"]
+ subgraph eks["EKS cluster 'airflow-ci' v1.34 (standard mode)"]
+ subgraph system["System node group — 2× t4g.medium on-demand"]
+ controller["ARC controller<br/>(arc-systems)"]
+ listener["ARC listener<br/>per scale set"]
+ karpenter["Karpenter v1.11<br/>(kube-system)"]
+ end
+ subgraph arcrunners["arc-runners namespace"]
+ secret["Secret github-config<br/>(GitHub App creds)"]
+ runner["Ephemeral runner pod<br/>runner + privileged dind sidecar<br/>requests 28 vCPU / 56 Gi · 1 job"]
+ end
+ node["Spot node — cheapest of<br/>c/m/r 6th-7th gen ≥ 8xlarge<br/>AL2023 · 200 Gi gp3 · IMDS hop 1"]
+ end
+ end
+ sqs["SQS queue<br/>airflow-ci-karpenter<br/>(spot interruptions)"]
+ s3[("S3 stash bucket<br/>airflow-ci-runner-cache-us-east-2")]
+ end
+
+ listener -->|"outbound long-poll<br/>(no inbound access)"| actions
+ listener -->|"job queued:<br/>request runner"| runner
+ secret -.-> listener
+ runner -.->|"Pending pod"| karpenter
+ karpenter -->|"launch / consolidate<br/>(WhenEmpty, 60s)"| node
+ runner ---|"scheduled on"| node
+ sqs --> karpenter
+ runner -->|"EKS Pod Identity<br/>(scoped S3 only)"| s3
+```
+
+Terraform (`terraform/eks/`) creates everything under **AWS**; helmfile
+(`helm/infra-helmfile.yaml`, `helm/ci-helmfile.yaml.gotmpl`) deploys everything inside
+**EKS**; the `github-config` secret is created once by hand at bootstrap.
+
+- **Terraform** (this repo, `terraform/eks/`): VPC, EKS cluster, system node group,
+ Karpenter controller IAM (Pod Identity), node IAM role, spot-interruption SQS queue.
+- **Helmfile** (this repo, `helm/`): Karpenter, ARC controller, runner scale sets,
+ Karpenter `NodePool`/`EC2NodeClass` CRs, namespaces.
+- **Airflow repo**: `runs-on: airflow-runner-amd-xl` on the Tests (AMD) job graph
+ (experiment branch; later behind the existing `selective_checks` runner seam for
+ canary-only routing).
+
+## Key decisions (and what was rejected)
+
+### EKS standard mode + self-managed Karpenter — not EKS Auto Mode
+
+Auto Mode was seriously considered (it removes Karpenter/AMI/addon management) and
+rejected for this workload:
+
+- Auto Mode nodes are locked-down Bottlerocket: SELinux enforcing, immutable root FS,
+ no SSH/SSM. CI runners are privileged docker-in-docker pods that run KinD
+ (Kubernetes-in-Docker-in-Docker) and need node sysctls (`fs.inotify.*`,
+ `nf_conntrack`, `fs.nr_open`) — exactly the class of workload that fights a locked
+ host. Standard AL2023 nodes with our own user data are the proven path.
+- Auto Mode adds a per-instance management fee (~10–12% of the instance price) on a
+ budget-constrained, compute-dominated bill.
+- We already need Helm/CRD management for ARC; Karpenter via the same helmfile is
+ marginal extra complexity.
+
+### Karpenter v1.11 — not cluster-autoscaler + 6 static node groups
+
+The previous design (this branch pre-rebase) used six per-size/arch managed node
+groups, ASG label/taint tags for scale-from-zero (`asg_tags.tf`), and
+cluster-autoscaler. Karpenter replaces all of it:
+
+- Spot-price-aware instance selection across families (the "instance diversity"
+ discussion in PR #59 becomes a one-line requirement list).
+- Scale-from-zero without ASG tag gymnastics → `asg_tags.tf` deleted.
+- Consolidation and native spot-interruption handling (SQS) built in.
+- One `NodePool` instead of N node groups; an ARM pool later is ~10 lines.
+
+### ARC runner scale sets 0.14.x — still the supported approach in 2026
+
+`gha-runner-scale-set` (Autoscaling Runner Scale Sets mode) remains GitHub's
+maintained ARC mode (latest chart 0.14.2, 2026-05). Ephemeral runners, JIT
+registration, no cert-manager/webhook dependency (that was legacy ARC). Auth via a
+**GitHub App** (recommended over PAT): for repo-level runners it needs
+*Administration: R/W* + *Metadata: R*; for org-level, *Self-hosted runners: R/W*.
+
+### Utilities: what's in, what's out
+
+| Utility | Verdict | Why |
+|---|---|---|
+| Karpenter | **In** | Node autoscaling; replaces CA + static node groups; spot handling. Cost: negligible (runs on system nodes). |
+| ARC (scale sets) | **In** | The runner integration itself. |
+| metrics-server | **Out** | ARC scales on the job queue, not resource metrics; no HPA anywhere. Add only if we later want `kubectl top`. |
+| cert-manager | **Out** (removed from branch) | Only legacy-mode ARC needed it for admission webhooks. Scale-set mode doesn't. |
+| EBS CSI driver | **Out** (removed from branch) | No PersistentVolumes: runner work dirs + dind storage are node-local ephemeral (200 GiB gp3 root via `EC2NodeClass`). |
+| cluster-autoscaler | **Out** (replaced) | See Karpenter. |
+| External Secrets Operator | **Out** | Exactly one static secret (GitHub App key). ESO adds a controller + still needs a backing store + IAM. `kubectl create secret` at bootstrap is simpler and rotated rarely. Revisit if secrets multiply. |
+| AWS Load Balancer Controller | **Out** | Nothing is exposed; ARC listeners do outbound long-poll only. No Service/Ingress of type LB. |
+
+### Networking: public subnets, no NAT — kept deliberately
+
+Nodes launch in public subnets with public IPs (as in the original PR, which removed
+the NAT gateway). A NAT gateway costs ~$395/yr + data processing; public IPv4 on
+ephemeral nodes costs $0.005/h only while nodes exist (≈$40/yr at our duty cycle).
+Security posture is acceptable because: security groups allow no inbound; runners
+only make outbound calls (GitHub long-poll, image pulls); nodes use IMDSv2 with hop
+limit 1, so **pods (i.e. CI jobs) cannot reach node IAM credentials**; and the
+node role carries only EKS/ECR/CNI policies.
+
+### Compute: one runner class now, right-sizing later
+
+Measured mix of a full Tests (AMD) run (duration-weighted minutes by required vCPU
+class): `2:787, 4:209, 8:179, 16:96, 32:22, 48:6`. The cost-optimal design is 2–3
+scale-set size classes, but that requires per-job `runs-on` routing in the Airflow
+repo. To keep the Airflow diff minimal for the experiment, we start with a **single
+class sized so no job regresses** vs the CodeBuild baseline (32+ vCPU nodes):
+
+- Scale set `airflow-runner-amd-xl`: requests **28 vCPU / 56 Gi** → Karpenter picks
+ the cheapest current-gen ≥8xlarge (c6a/c7a/m6a/m7a/r6a/r7a, 32 vCPU) spot node;
+ one runner pod per node (requests-based, no anti-affinity hack).
+- Spot first with automatic on-demand fallback (`capacity-type: [spot, on-demand]`).
+ Consolidation is `WhenEmpty` only — running jobs are never evicted to repack.
+- maxRunners 100 (measured burst peak: 77 concurrent on the reduced run, ~163 on a
+ full canary; raise when canary-scale traffic is routed here).
+
+Cost at measured volume: ~21.7 node-hours per full run × ~$0.55–0.60/h (8xlarge
+spot) ≈ **$12–13/run** vs $104 on CodeBuild — before phase-2 right-sizing, which
+takes it to ~$4–5.
+
+**ARM later: yes, worth it.** Graviton spot is 10–25% cheaper per vCPU and the ARM
+canary exists. It's one extra `NodePool` (arch: arm64) + one scale set + an ARM
+runner image tag. Deliberately out of scope until AMD is proven.
+
+### dind configuration carries the CodeBuild lessons
+
+The runner pods define an explicit dind sidecar (not the chart's default) so dockerd
+starts with flags that fix real failures seen on CodeBuild:
+
+- `--registry-mirror=https://mirror.gcr.io` — Docker Hub anonymous pull limits on
+ shared cloud egress IPs (also covers in-container `testcontainers` pulls).
+- `--default-address-pool base=172.16.0.0/14,size=24` — parallel breeze compose
+ networks exhaust Docker's default ~31 subnets.
+- `--default-ulimit nofile=65536:1048576` — venv-heavy tests exhaust fds; soft cap
+ kept moderate to avoid the close()-loop spawn trap.
+- cgroup-v2 delegation for KinD: `docker:dind`'s entrypoint handles this correctly
+ (the CodeBuild failure existed precisely because its dockerd was *not* dind).
+- Node sysctls (`fs.inotify.max_user_watches/instances`, `nf_conntrack_tcp_be_liberal`,
+ `fs.nr_open`, `fs.file-max`) are set in `EC2NodeClass` user data.
+
+### Terraform layout: one root module, S3 backend
+
+Single root module at `terraform/eks/` (no premature module split — the AWS-side
+footprint is ~5 resources plus two registry modules). Existing S3 backend
+(`airflow-ci-tfstate` / DynamoDB `TerraformLocks`, us-east-2) is kept for the real
+deployment; it still holds the **stale 2024 state** of the deleted cluster (the
+`airflow-vpc` 10.1.0.0/16 VPC still exists in the account!) — reconcile before the
+first real apply (import-or-destroy; see README). Local iteration uses a
+`*_override.tf` local backend so `plan` runs without touching shared state. The
+provider no longer hardcodes `profile = "airflow"` — use `AWS_PROFILE`/env, which
+also works for a future GitHub Actions plan/apply workflow via OIDC role assumption.
+
+Versions: Terraform ≥ 1.9, AWS provider ~> 6.0, `terraform-aws-modules/eks` ~> 21.24
+(EKS 1.34, standard support to 2027-12), `terraform-aws-modules/vpc` ~> 6.6,
+Karpenter 1.11.3, ARC chart 0.14.2.
+
+### Security & trust boundary
+
+- Runners serve **trusted refs only**: scheduled/canary runs and (later) explicitly
+ label-gated committer PRs. Fork PRs stay on GitHub-hosted runners. For the org
+ deployment, the scale sets sit in a dedicated runner group restricted to
+ apache/airflow with "public repositories: selected workflows" disabled.
+- GitHub App over PAT: scoped, auditable, revocable; key lives only in the
+ `arc-runners` namespace secret.
+- Ephemeral runners: one job per pod, pod deleted after the job — no state bleed.
+- Jobs can't reach node IAM credentials (IMDS hop limit 1); the node role has no
+ access beyond EKS basics; the AWS account contains no Airflow production secrets.
+- Cluster API endpoint is public + access entries (admin = cluster creator +
+ explicitly added PMC operators); no aws-auth ConfigMap.
+
+## Scaling behavior
+
+1. Job queued on GitHub → ARC listener (long-poll) sees it, requests a runner pod.
+2. Pod is Pending (no capacity) → Karpenter launches the cheapest fitting spot
+ instance (~60–90 s to Ready including image pull).
+3. Job runs in the ephemeral runner (dind sidecar provides Docker).
+4. Pod completes and is deleted → node goes empty → Karpenter consolidates it away
+ after 60 s. Idle cluster = 2× t4g.medium + control plane ≈ **$1.6k/yr fixed**.
+5. Spot interruption → SQS → Karpenter drains/replaces; the interrupted job fails
+ and is retried by the workflow's normal retry/re-run mechanisms (acceptable for
+ canary; pin the pool to on-demand if measured interruption rates hurt).
+
+## Costs (measured workload, official prices)
+
+| | CodeBuild (today) | EKS this design | EKS after right-sizing |
+|---|---|---|---|
+| Full Tests (AMD) run | $104 | ~$12–13 | ~$4–5 |
+| Scheduled-only, 2 AMD + 2 ARM per day, /yr | ~$135k+ | ~$20k | ~$9–11k |
+| Fixed (cluster + system nodes + misc) | — | ~$1.8–2.1k/yr | same |
+
+## What changes in the Airflow repo (experiment branch)
+
+Only the Tests (AMD) job graph: `runs-on` swapped from the CodeBuild label array to
+`airflow-runner-amd-xl`, and the CodeBuild-specific host guards extended to cover
+the ARC runner environment. Everything else (selective checks, breeze, tests) is
+untouched. The long-term mechanism is the existing `selective_checks` runner seam
+(static labels work with ARC, unlike CodeBuild's per-run label), flipped for canary
+runs only.
diff --git a/helm/ci-helmfile.yaml b/helm/ci-helmfile.yaml.gotmpl
similarity index 76%
rename from helm/ci-helmfile.yaml
rename to helm/ci-helmfile.yaml.gotmpl
index e1da5a0..c707c9f 100644
--- a/helm/ci-helmfile.yaml
+++ b/helm/ci-helmfile.yaml.gotmpl
@@ -20,19 +20,26 @@ environments:
values:
- ./values/gha-runner-scale-sets/runners.yaml
+# helmfile v1 requires environments and releases in separate YAML documents.
+---
releases:
- name: actions-runner-controller
chart: oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
- namespace: ci
- version: 0.9.3
+ namespace: arc-systems
+ version: 0.14.2
values:
- ./values/gha-runner-scale-set-controller.yaml
{{- range $key, $value := .Values.runnerScaleSets }}
- name: {{ $key }}
chart: oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
- namespace: ci
- version: 0.9.3
+ namespace: arc-runners
+ version: 0.14.2
+ needs:
+ - arc-systems/actions-runner-controller
+ # AutoscalingRunnerSet CRD ships with the controller chart above — skip
+ # upfront validation so a fresh cluster can install in one pass.
+ disableValidationOnInstall: true
values:
- ./values/gha-runner-scale-sets/common.yaml.gotmpl
{{- end }}
diff --git a/helm/cluster-config/templates/namespaces/infra.yaml b/helm/cluster-config/templates/namespaces/arc.yaml
similarity index 85%
rename from helm/cluster-config/templates/namespaces/infra.yaml
rename to helm/cluster-config/templates/namespaces/arc.yaml
index f6bd908..50450f3 100644
--- a/helm/cluster-config/templates/namespaces/infra.yaml
+++ b/helm/cluster-config/templates/namespaces/arc.yaml
@@ -18,9 +18,14 @@
*/}}
################################################################################
-# infra namespace where we create the infrastructure resources
+# arc-systems: ARC controller + listeners; arc-runners: runner pods + secrets
################################################################################
apiVersion: v1
kind: Namespace
metadata:
- name: infra
+ name: arc-systems
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+ name: arc-runners
diff --git a/helm/cluster-config/templates/namespaces/ci.yaml b/helm/cluster-config/templates/namespaces/ci.yaml
deleted file mode 100644
index 123ddc1..0000000
--- a/helm/cluster-config/templates/namespaces/ci.yaml
+++ /dev/null
@@ -1,26 +0,0 @@
-{{/*
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-*/}}
-
-################################################################################
-# ci namespace where we create the CI resources
-################################################################################
-apiVersion: v1
-kind: Namespace
-metadata:
- name: ci
diff --git a/helm/infra-helmfile.yaml b/helm/infra-helmfile.yaml
index d8c49e5..100dd84 100644
--- a/helm/infra-helmfile.yaml
+++ b/helm/infra-helmfile.yaml
@@ -15,38 +15,26 @@
# specific language governing permissions and limitations
# under the License.
---
-repositories:
- - name: jetstack
- url: https://charts.jetstack.io
- - name: autoscaler
- url: https://kubernetes.github.io/autoscaler
- - name: aws-ebs-csi-driver
- url: https://kubernetes-sigs.github.io/aws-ebs-csi-driver
-
releases:
- name: cluster-config
chart: ./cluster-config
namespace: kube-system
- # https://github.com/kubernetes-sigs/aws-ebs-csi-driver/tree/master/charts/aws-ebs-csi-driver
- - name: aws-ebs-csi-driver
- chart: aws-ebs-csi-driver/aws-ebs-csi-driver
+ # https://karpenter.sh/docs/getting-started/
+ # IAM/SQS for Karpenter comes from terraform/eks/karpenter.tf (Pod Identity).
+ - name: karpenter
+ chart: oci://public.ecr.aws/karpenter/karpenter
namespace: kube-system
+ version: 1.11.3
values:
- - ./values/ebs-csi-dirver.yaml
-
- # https://cert-manager.io/docs/installation/helm/
- - name: cert-manager
- chart: jetstack/cert-manager
- namespace: infra
- version: v1.15.3
- values:
- - ./values/cert-manager.yaml
+ - ./values/karpenter.yaml
- # https://github.com/kubernetes/autoscaler/tree/master/charts/cluster-autoscaler
- - name: autoscaler
- chart: autoscaler/cluster-autoscaler
- namespace: infra
- version: 9.37.0
- values:
- - ./values/cluster-autoscaler.yaml
+ # NodePool/EC2NodeClass CRs — depends on the Karpenter CRDs.
+ - name: karpenter-resources
+ chart: ./karpenter-resources
+ namespace: kube-system
+ needs:
+ - kube-system/karpenter
+ # On a fresh cluster the CRDs don't exist until the karpenter release above
+ # installs, so the upfront diff/validation of this release must be skipped.
+ disableValidationOnInstall: true
diff --git a/helm/values/ebs-csi-dirver.yaml b/helm/karpenter-resources/Chart.yaml
similarity index 83%
rename from helm/values/ebs-csi-dirver.yaml
rename to helm/karpenter-resources/Chart.yaml
index 7f8f0d0..e33e60d 100644
--- a/helm/values/ebs-csi-dirver.yaml
+++ b/helm/karpenter-resources/Chart.yaml
@@ -15,7 +15,8 @@
# specific language governing permissions and limitations
# under the License.
---
-controller:
- serviceAccount:
- annotations:
- eks.amazonaws.com/role-arn: arn:aws:iam::827901512104:role/ebs-csi-driver
+apiVersion: v2
+name: karpenter-resources
+description: Karpenter NodePools and EC2NodeClasses for the Airflow CI runners
+type: application
+version: 0.1.0
diff --git a/helm/karpenter-resources/templates/ec2nodeclass.yaml b/helm/karpenter-resources/templates/ec2nodeclass.yaml
new file mode 100644
index 0000000..b5058dd
--- /dev/null
+++ b/helm/karpenter-resources/templates/ec2nodeclass.yaml
@@ -0,0 +1,71 @@
+{{/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+*/}}
+apiVersion: karpenter.k8s.aws/v1
+kind: EC2NodeClass
+metadata:
+ name: ci
+spec:
+ amiSelectorTerms:
+ - alias: al2023@latest
+ role: {{ .Values.nodeIamRoleName }}
+ subnetSelectorTerms:
+ - tags:
+ karpenter.sh/discovery: {{ .Values.clusterName }}
+ securityGroupSelectorTerms:
+ - tags:
+ karpenter.sh/discovery: {{ .Values.clusterName }}
+ associatePublicIPAddress: true
+ metadataOptions:
+ httpTokens: required
+ # Hop limit 1 keeps node IAM credentials unreachable from the runner
+ # containers (CI jobs must not get AWS access via IMDS).
+ httpPutResponseHopLimit: 1
+ blockDeviceMappings:
+ - deviceName: /dev/xvda
+ ebs:
+ volumeSize: {{ .Values.nodeVolumeSize }}
+ volumeType: gp3
+ iops: 4000
+ throughput: 250
+ encrypted: true
+ deleteOnTermination: true
+ userData: |
+ MIME-Version: 1.0
+ Content-Type: multipart/mixed; boundary="BOUNDARY"
+
+ --BOUNDARY
+ Content-Type: text/x-shellscript; charset="us-ascii"
+
+ #!/bin/bash
+ # Kernel limits for KinD clusters and fd-heavy test suites; GitHub-hosted
+ # runners ship these, stock AL2023 does not.
+ modprobe nf_conntrack || true
+ sysctl -w fs.inotify.max_user_watches=524288
+ sysctl -w fs.inotify.max_user_instances=512
+ sysctl -w net.netfilter.nf_conntrack_tcp_be_liberal=1 || true
+ sysctl -w fs.nr_open=2097152
+ sysctl -w fs.file-max=2097152
+ # Disable core dumps: AL2023's default systemd-coredump handler is
+ # kernel-global, so a SEGV inside any container dumps the whole process
+ # through the node's journald first — under test-suite disk load that
+ # takes >60s and broke task-sdk's test_exit_by_signal[segv] (GitHub's
+ # Ubuntu runners short-circuit via apport instead). CI nodes are
+ # ephemeral; nobody reads their dumps.
+ sysctl -w "kernel.core_pattern=|/bin/false"
+ --BOUNDARY--
diff --git a/helm/karpenter-resources/templates/nodepool-amd-shared.yaml b/helm/karpenter-resources/templates/nodepool-amd-shared.yaml
new file mode 100644
index 0000000..b2f2eab
--- /dev/null
+++ b/helm/karpenter-resources/templates/nodepool-amd-shared.yaml
@@ -0,0 +1,65 @@
+{{/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+*/}}
+{{/*
+ On-demand pool for the bin-packed small/medium runner classes. Shared nodes
+ concentrate many jobs, so a single spot reclaim kills them all at once and a
+ rerun re-packs the same jobs into the same vulnerable shape (measured
+ 2026-07-27/28: reclaim storms of 5-57 events wiped whole retry batches
+ repeatedly; the on-demand premium is ~$1.5-2 per run). The xl class stays on
+ the spot-first `ci-amd` pool: one job per node, so reclaims stay cheap,
+ independent retries.
+*/}}
+apiVersion: karpenter.sh/v1
+kind: NodePool
+metadata:
+ name: ci-amd-shared
+spec:
+ template:
+ metadata:
+ labels:
+ airflow-ci/pool: amd-shared
+ spec:
+ nodeClassRef:
+ group: karpenter.k8s.aws
+ kind: EC2NodeClass
+ name: ci
+ taints:
+ - key: airflow-ci/runner
+ effect: NoSchedule
+ requirements:
+ - key: kubernetes.io/arch
+ operator: In
+ values: ["amd64"]
+ - key: kubernetes.io/os
+ operator: In
+ values: ["linux"]
+ - key: karpenter.sh/capacity-type
+ operator: In
+ values: ["on-demand"]
+ - key: karpenter.k8s.aws/instance-family
+ operator: In
+ values: ["c6a", "c7a", "m6a", "m7a", "r6a", "r7a", "c6i", "c7i", "m6i", "m7i"]
+ - key: karpenter.k8s.aws/instance-cpu
+ operator: In
+ values: ["8", "16", "32", "48"]
+ disruption:
+ consolidationPolicy: WhenEmpty
+ consolidateAfter: 60s
+ limits:
+ cpu: {{ .Values.amdSharedPoolCpuLimit | quote }}
diff --git a/helm/karpenter-resources/templates/nodepool-amd.yaml b/helm/karpenter-resources/templates/nodepool-amd.yaml
new file mode 100644
index 0000000..b053dba
--- /dev/null
+++ b/helm/karpenter-resources/templates/nodepool-amd.yaml
@@ -0,0 +1,63 @@
+{{/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+*/}}
+apiVersion: karpenter.sh/v1
+kind: NodePool
+metadata:
+ name: ci-amd
+spec:
+ template:
+ metadata:
+ labels:
+ airflow-ci/pool: amd
+ spec:
+ nodeClassRef:
+ group: karpenter.k8s.aws
+ kind: EC2NodeClass
+ name: ci
+ # Only runner pods (which tolerate this) land on these nodes, so they
+ # never pin cluster infra and can be reclaimed the moment a job ends.
+ taints:
+ - key: airflow-ci/runner
+ effect: NoSchedule
+ requirements:
+ - key: kubernetes.io/arch
+ operator: In
+ values: ["amd64"]
+ - key: kubernetes.io/os
+ operator: In
+ values: ["linux"]
+ # Spot first; Karpenter falls back to on-demand when spot is unavailable.
+ - key: karpenter.sh/capacity-type
+ operator: In
+ values: ["spot", "on-demand"]
+ - key: karpenter.k8s.aws/instance-family
+ operator: In
+ values: ["c6a", "c7a", "m6a", "m7a", "r6a", "r7a", "c6i", "c7i", "m6i", "m7i"]
+ # Small enough sizes for the small/medium runner classes to get
+ # right-sized (or bin-packed) nodes; xl pods still land on >= 32 vCPU
+ # because their 28-vCPU request cannot fit anything smaller.
+ - key: karpenter.k8s.aws/instance-cpu
+ operator: In
+ values: ["8", "16", "32", "48"]
+ disruption:
+ # Never repack running CI jobs; only reap empty nodes.
+ consolidationPolicy: WhenEmpty
+ consolidateAfter: 60s
+ limits:
+ cpu: {{ .Values.amdPoolCpuLimit | quote }}
diff --git a/helm/karpenter-resources/templates/nodepool-arm-shared.yaml b/helm/karpenter-resources/templates/nodepool-arm-shared.yaml
new file mode 100644
index 0000000..b1c402b
--- /dev/null
+++ b/helm/karpenter-resources/templates/nodepool-arm-shared.yaml
@@ -0,0 +1,65 @@
+{{/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+*/}}
+{{/*
+ On-demand pool for the bin-packed small/medium runner classes. Shared nodes
+ concentrate many jobs, so a single spot reclaim kills them all at once and a
+ rerun re-packs the same jobs into the same vulnerable shape (measured
+ 2026-07-27/28: reclaim storms of 5-57 events wiped whole retry batches
+ repeatedly; the on-demand premium is ~$1.5-2 per run). The xl class stays on
+ the spot-first `ci-arm` pool: one job per node, so reclaims stay cheap,
+ independent retries.
+*/}}
+apiVersion: karpenter.sh/v1
+kind: NodePool
+metadata:
+ name: ci-arm-shared
+spec:
+ template:
+ metadata:
+ labels:
+ airflow-ci/pool: arm-shared
+ spec:
+ nodeClassRef:
+ group: karpenter.k8s.aws
+ kind: EC2NodeClass
+ name: ci
+ taints:
+ - key: airflow-ci/runner
+ effect: NoSchedule
+ requirements:
+ - key: kubernetes.io/arch
+ operator: In
+ values: ["arm64"]
+ - key: kubernetes.io/os
+ operator: In
+ values: ["linux"]
+ - key: karpenter.sh/capacity-type
+ operator: In
+ values: ["on-demand"]
+ - key: karpenter.k8s.aws/instance-family
+ operator: In
+ values: ["c6g", "c7g", "c8g", "m6g", "m7g", "m8g", "r6g", "r7g", "r8g"]
+ - key: karpenter.k8s.aws/instance-cpu
+ operator: In
+ values: ["8", "16", "32", "48"]
+ disruption:
+ consolidationPolicy: WhenEmpty
+ consolidateAfter: 60s
+ limits:
+ cpu: {{ .Values.armSharedPoolCpuLimit | quote }}
diff --git a/helm/karpenter-resources/templates/nodepool-arm.yaml b/helm/karpenter-resources/templates/nodepool-arm.yaml
new file mode 100644
index 0000000..2d6e763
--- /dev/null
+++ b/helm/karpenter-resources/templates/nodepool-arm.yaml
@@ -0,0 +1,63 @@
+{{/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+*/}}
+apiVersion: karpenter.sh/v1
+kind: NodePool
+metadata:
+ name: ci-arm
+spec:
+ template:
+ metadata:
+ labels:
+ airflow-ci/pool: arm
+ spec:
+ nodeClassRef:
+ group: karpenter.k8s.aws
+ kind: EC2NodeClass
+ name: ci
+ # Only runner pods (which tolerate this) land on these nodes, so they
+ # never pin cluster infra and can be reclaimed the moment a job ends.
+ taints:
+ - key: airflow-ci/runner
+ effect: NoSchedule
+ requirements:
+ - key: kubernetes.io/arch
+ operator: In
+ values: ["arm64"]
+ - key: kubernetes.io/os
+ operator: In
+ values: ["linux"]
+ # Spot first; Karpenter falls back to on-demand when spot is unavailable.
+ - key: karpenter.sh/capacity-type
+ operator: In
+ values: ["spot", "on-demand"]
+ - key: karpenter.k8s.aws/instance-family
+ operator: In
+ values: ["c6g", "c7g", "c8g", "m6g", "m7g", "m8g", "r6g", "r7g", "r8g"]
+ # Small enough sizes for the small/medium runner classes to get
+ # right-sized (or bin-packed) nodes; xl pods still land on >= 32 vCPU
+ # because their 28-vCPU request cannot fit anything smaller.
+ - key: karpenter.k8s.aws/instance-cpu
+ operator: In
+ values: ["8", "16", "32", "48"]
+ disruption:
+ # Never repack running CI jobs; only reap empty nodes.
+ consolidationPolicy: WhenEmpty
+ consolidateAfter: 60s
+ limits:
+ cpu: {{ .Values.armPoolCpuLimit | quote }}
diff --git a/helm/values/cluster-autoscaler.yaml b/helm/karpenter-resources/values.yaml
similarity index 51%
copy from helm/values/cluster-autoscaler.yaml
copy to helm/karpenter-resources/values.yaml
index 2b4bf57..48c6b18 100644
--- a/helm/values/cluster-autoscaler.yaml
+++ b/helm/karpenter-resources/values.yaml
@@ -15,21 +15,21 @@
# specific language governing permissions and limitations
# under the License.
---
-cloudProvider: aws
-awsRegion: us-east-2
+clusterName: airflow-ci
-autoDiscovery:
- clusterName: "airflow"
+# Must match node_iam_role_name in terraform/eks/karpenter.tf.
+nodeIamRoleName: airflow-ci-runner-node
-rbac:
- create: true
- serviceAccount:
- create: true
- name: "cluster-autoscaler"
- annotations:
- eks.amazonaws.com/role-arn: arn:aws:iam::827901512104:role/eks-autoscaler-role
+# Root volume for runner nodes: CI image + KinD images + parallel compose
+# stacks need far more than the 20Gi AL2023 default (CodeBuild boxes had 256GB).
+nodeVolumeSize: 200Gi
-extraArgs:
- scan-interval: 10s
- scale-down-delay-after-add: 5m
- scale-down-unneeded-time: 5m
+# Upper bound on total simultaneously provisioned runner capacity
+# (~100 concurrent 32-vCPU runner nodes).
+amdPoolCpuLimit: "3200"
+# On-demand shared pool (small/medium classes): sized for the measured
+# full-matrix peak (~60 nodes averaging well under 48 vCPU each).
+amdSharedPoolCpuLimit: "2000"
+# ARM (Graviton) pools mirror the AMD split; experiment-sized like amd was.
+armPoolCpuLimit: "3200"
+armSharedPoolCpuLimit: "2000"
diff --git a/helm/values/cert-manager.yaml b/helm/values/cert-manager.yaml
deleted file mode 100644
index be548e5..0000000
--- a/helm/values/cert-manager.yaml
+++ /dev/null
@@ -1,20 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
----
-installCRDs: true
-nodeSelector:
- node-type: default
diff --git a/helm/values/gha-runner-scale-set-controller.yaml b/helm/values/gha-runner-scale-set-controller.yaml
index 63ab994..1ed8177 100644
--- a/helm/values/gha-runner-scale-set-controller.yaml
+++ b/helm/values/gha-runner-scale-set-controller.yaml
@@ -18,8 +18,7 @@
replicaCount: 1
labels:
- type: ci
app: gha-runner-scale-set-controller
-nodeSelector:
- node-type: default
+# Watch all namespaces (scale sets live in arc-runners); no nodeSelector needed:
+# runner nodes are tainted, so the controller lands on the system node group.
diff --git a/helm/values/gha-runner-scale-sets/common.yaml.gotmpl b/helm/values/gha-runner-scale-sets/common.yaml.gotmpl
index 9ed5238..1b23173 100644
--- a/helm/values/gha-runner-scale-sets/common.yaml.gotmpl
+++ b/helm/values/gha-runner-scale-sets/common.yaml.gotmpl
@@ -17,45 +17,134 @@
---
{{ $config := index .Values.runnerScaleSets .Release.Name }}
-githubConfigUrl: https://github.com/apache
+githubConfigUrl: {{ .Values.githubConfigUrl }}
+# Pre-created secret in the arc-runners namespace holding the GitHub App
+# credentials (github_app_id / github_app_installation_id / github_app_private_key).
githubConfigSecret: github-config
-runnerGroup: airflow-ci
-
minRunners: {{ $config.minRunners }}
maxRunners: {{ $config.maxRunners }}
-containerMode:
- type: "dind"
-
+# The dind sidecar is spelled out (instead of `containerMode: dind`) so dockerd
+# gets flags that fix real failures measured on the CodeBuild runners:
+# --registry-mirror Docker Hub anonymous pull limits on shared cloud
+# egress IPs (also covers in-container testcontainers
+# pulls that cannot use host credentials).
+# --default-address-pool parallel breeze compose stacks exhaust Docker's
+# default ~31 network pools; /14 carved into /24s
+# gives 1024 networks. Must not overlap the VPC CIDR
+# (10.1.0.0/16).
+# --default-ulimit nofile fd-heavy venv tests need >1024; the soft cap stays
+# at 65536 because processes that close(0..soft) on
+# spawn make a high soft limit a performance trap.
template:
+ metadata:
+ annotations:
+ # Karpenter must never drain a node out from under a running CI job —
+ # not for drift (e.g. EC2NodeClass changes) and not for consolidation.
+ # Involuntary events (spot reclaim) are unaffected.
+ karpenter.sh/do-not-disrupt: "true"
spec:
+ # CI jobs must not see this cluster's credentials: with the token mounted,
+ # kubernetes clients inside tests prefer in-cluster config over the KinD
+ # kubeconfig and hit the EKS API as the runner SA (403s broke the K8S
+ # system-test teardowns). EKS Pod Identity (S3 stash) uses its own
+ # separately-projected token and is unaffected.
+ automountServiceAccountToken: false
+ initContainers:
+ - name: init-dind-externals
+ image: {{ .Values.image }}
+ command: ["cp", "-r", "/home/runner/externals/.", "/home/runner/tmpDir/"]
+ volumeMounts:
+ - name: dind-externals
+ mountPath: /home/runner/tmpDir
+ - name: dind
+ image: docker:dind
+ args:
+ - dockerd
+ - --host=unix:///var/run/docker.sock
+ - --group=$(DOCKER_GROUP_GID)
+ - --registry-mirror=https://mirror.gcr.io
+ - --default-address-pool=base=172.16.0.0/14,size=24
+ - --default-ulimit=nofile=65536:1048576
+ # Classic image store (overlay2), matching GitHub-hosted runners.
+ # The containerd store (dind default) breaks `kind load docker-image`:
+ # docker save emits OCI manifests whose blobs ctr import --digests
+ # can't resolve ("content digest ... not found").
+ - --feature=containerd-snapshotter=false
+ env:
+ - name: DOCKER_GROUP_GID
+ value: "123"
+ securityContext:
+ privileged: true
+ restartPolicy: Always
+ startupProbe:
+ exec:
+ command: ["docker", "info"]
+ periodSeconds: 5
+ failureThreshold: 24
+ volumeMounts:
+ - name: work
+ mountPath: /home/runner/_work
+ - name: dind-sock
+ mountPath: /var/run
+ - name: dind-externals
+ mountPath: /home/runner/externals
+ # /tmp is shared with the runner (see runner container) so that
+ # `docker run -v /tmp/...` host paths resolve — the daemon lives
+ # here, not in the runner container.
+ - name: shared-tmp
+ mountPath: /tmp
containers:
- name: runner
image: {{ .Values.image }}
- command: [ "/home/runner/run.sh" ]
+ command: ["/home/runner/run.sh"]
+ env:
+ - name: DOCKER_HOST
+ value: unix:///var/run/docker.sock
+ - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
+ value: "120"
+ # Lets airflow-repo CI scripts detect the self-hosted K8s runner
+ # (guards for host-specific steps like the /mnt docker relocation).
+ - name: AIRFLOW_SELF_HOSTED_RUNNER
+ value: "eks"
+ # Class-sized xdist worker cap; run_unit_tests.sh only defaults
+ # PARALLELISM when unset, so this wins over its self-hosted fallback.
+ - name: PARALLELISM
+ value: {{ $config.parallelism | quote }}
+ resources:
+ requests:
+ cpu: {{ $config.cpu | quote }}
+ memory: {{ $config.memory }}
+ volumeMounts:
+ - name: work
+ mountPath: /home/runner/_work
+ - name: dind-sock
+ mountPath: /var/run
+ # Shared with the dind sidecar: tests bind-mount pytest tmpdirs
+ # (docker run -v /tmp/...) and the daemon resolves those paths in
+ # ITS filesystem. On GitHub-hosted runners daemon and job share one
+ # host; this emptyDir recreates that contract for /tmp.
+ - name: shared-tmp
+ mountPath: /tmp
+ volumes:
+ - name: work
+ emptyDir: {}
+ - name: dind-sock
+ emptyDir: {}
+ - name: dind-externals
+ emptyDir: {}
+ - name: shared-tmp
+ emptyDir: {}
nodeSelector:
- node-type: gha-runners
- size: {{ $config.size }}
- arch: {{ $config.arch }}
+ airflow-ci/pool: {{ $config.pool }}
tolerations:
- - key: node-type
- operator: Equal
- value: gha-runners
+ - key: airflow-ci/runner
+ operator: Exists
effect: NoSchedule
- affinity:
- podAntiAffinity:
- requiredDuringSchedulingIgnoredDuringExecution:
- - labelSelector:
- matchExpressions:
- - key: actions.github.com/scale-set-name
- operator: In
- values:
- - {{ .Release.Name }}
- topologyKey: kubernetes.io/hostname
controllerServiceAccount:
name: actions-runner-controller-gha-rs-controller
- namespace: {{ .Release.Namespace }}
+ namespace: arc-systems
diff --git a/helm/values/gha-runner-scale-sets/runners.yaml b/helm/values/gha-runner-scale-sets/runners.yaml
index 3c475fe..4d86536 100644
--- a/helm/values/gha-runner-scale-sets/runners.yaml
+++ b/helm/values/gha-runner-scale-sets/runners.yaml
@@ -15,35 +15,70 @@
# specific language governing permissions and limitations
# under the License.
---
-image: ghcr.io/apache/airflow-ci-infra/actions-runner:20240902-rc1
+# Always a date-pinned tag, never :latest — an unreviewed image change must not
+# be able to break every canary at once. Rebuild monthly or when ARC's minimum
+# runner version forces it.
+# Fork-experiment override (public package on the fork owner's ghcr); switch to
+# ghcr.io/apache/airflow-ci-infra/actions-runner:<tag> for the production deploy.
+image: ghcr.io/shahar1/airflow-ci-actions-runner:20260731.2
+
+# Repo URL for the fork experiment; switch to https://github.com/apache
+# (org-level, with a restricted runner group) for the production deployment.
+githubConfigUrl: https://github.com/shahar1/airflow
+
+# Three size classes, assigned per job from the measured per-job peaks of the
+# 2026-07-26 go/no-go runs (airflow repo files/eks-class-mapping.md): each job's
+# runs-on picks the smallest class fitting peak x 1.3 headroom on both CPU and
+# memory (Non-DB providers is memory-bound at 35 GiB, hence xl). The xl request
+# fills a whole 32-vCPU node (one job = one node); small/medium pods bin-pack
+# when Karpenter batches several pending jobs onto one larger spot node.
+# parallelism feeds the PARALLELISM env (xdist worker cap in run_unit_tests.sh)
+# so test parallelism matches the class instead of oversubscribing small nodes.
runnerScaleSets:
- arc-small-amd:
- minRunners: 0
- maxRunners: 30
- size: small
- arch: x64
- arc-medium-amd:
- minRunners: 0
- maxRunners: 30
- size: medium
- arch: x64
- arc-large-amd:
- minRunners: 0
- maxRunners: 30
- size: large
- arch: x64
- arc-small-arm:
- minRunners: 0
- maxRunners: 30
- size: small
- arch: arm64
- arc-medium-arm:
- minRunners: 0
- maxRunners: 30
- size: medium
- arch: arm64
- arc-large-arm:
- minRunners: 0
- maxRunners: 30
- size: large
- arch: arm64
+ # small/medium bin-pack onto shared nodes -> on-demand pool (a spot reclaim
+ # of a shared node kills every job on it and reruns re-pack the same batch;
+ # see nodepool-amd-shared.yaml).
+ airflow-runner-amd-small:
+ minRunners: 0
+ maxRunners: 60
+ pool: amd-shared
+ cpu: "4"
+ memory: 16Gi
+ parallelism: "4"
+ airflow-runner-amd-medium:
+ minRunners: 0
+ maxRunners: 80
+ pool: amd-shared
+ cpu: "8"
+ memory: 32Gi
+ parallelism: "8"
+ airflow-runner-amd-xl:
+ minRunners: 0
+ maxRunners: 100
+ pool: amd
+ cpu: "28"
+ memory: 56Gi
+ parallelism: "16"
+ # ARM (Graviton) classes mirror the AMD layout; the image tag is a
+ # multi-arch manifest so the same reference serves both architectures.
+ airflow-runner-arm-small:
+ minRunners: 0
+ maxRunners: 60
+ pool: arm-shared
+ cpu: "4"
+ memory: 16Gi
+ parallelism: "4"
+ airflow-runner-arm-medium:
+ minRunners: 0
+ maxRunners: 80
+ pool: arm-shared
+ cpu: "8"
+ memory: 32Gi
+ parallelism: "8"
+ airflow-runner-arm-xl:
+ minRunners: 0
+ maxRunners: 100
+ pool: arm
+ cpu: "28"
+ memory: 56Gi
+ parallelism: "16"
diff --git a/helm/values/cluster-autoscaler.yaml b/helm/values/karpenter.yaml
similarity index 68%
rename from helm/values/cluster-autoscaler.yaml
rename to helm/values/karpenter.yaml
index 2b4bf57..4c5780f 100644
--- a/helm/values/cluster-autoscaler.yaml
+++ b/helm/values/karpenter.yaml
@@ -15,21 +15,17 @@
# specific language governing permissions and limitations
# under the License.
---
-cloudProvider: aws
-awsRegion: us-east-2
+settings:
+ clusterName: airflow-ci
+ # Must match queue_name in terraform/eks/karpenter.tf.
+ interruptionQueue: airflow-ci-karpenter
-autoDiscovery:
- clusterName: "airflow"
+# IAM comes from the EKS Pod Identity association created by Terraform —
+# no serviceAccount role annotation needed.
-rbac:
- create: true
- serviceAccount:
- create: true
- name: "cluster-autoscaler"
- annotations:
- eks.amazonaws.com/role-arn: arn:aws:iam::827901512104:role/eks-autoscaler-role
-
-extraArgs:
- scan-interval: 10s
- scale-down-delay-after-add: 5m
- scale-down-unneeded-time: 5m
+# Fits the 2x t4g.medium system nodes.
+controller:
+ resources:
+ requests:
+ cpu: 500m
+ memory: 768Mi
diff --git a/runner/Dockerfile b/runner/Dockerfile
index 683028a..6fc9126 100644
--- a/runner/Dockerfile
+++ b/runner/Dockerfile
@@ -19,17 +19,97 @@ FROM ghcr.io/actions/actions-runner:latest
USER root
+# Host-level tools the airflow CI workflows expect on a runner — the target
+# contract is GitHub's hosted ubuntu-24.04 image, restricted to what the CI
+# graph actually invokes (pip, uv-via-pip, aws, parallel, docker CLI). Jobs
+# also run `sudo apt-get install ...` themselves (as on GitHub-hosted
+# runners), so the runner user gets passwordless sudo instead of us baking in
+# every package.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
- ca-certificates curl nodejs npm wget unzip vim git jq build-essential netcat subversion \
+ sudo ca-certificates curl wget zip unzip zstd git jq parallel build-essential netcat-openbsd \
libssl-dev libffi-dev python3 python3-venv python3-dev python3-pip python-is-python3 \
- && npm install -g yarn \
&& install -m 0755 -d /etc/apt/keyrings \
&& curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc \
&& chmod a+r /etc/apt/keyrings/docker.asc \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null \
&& apt-get update \
- && apt-get install -y docker-compose-plugin
+ && apt-get install -y --no-install-recommends docker-ce-cli docker-compose-plugin docker-buildx-plugin \
+ && rm -rf /var/lib/apt/lists/* \
+ && echo "runner ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/runner \
+ && chmod 0440 /etc/sudoers.d/runner
-USER runner
\ No newline at end of file
+# The actions-runner base ships a malformed tz config (/etc/timezone = "/UTC",
+# localtime -> zoneinfo//UTC) which breaks local-timezone detection with
+# 'ZoneInfo keys may not be absolute paths' (seen in shared timezones tests).
+# Normalize to what GitHub-hosted runners have.
+RUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime && echo "Etc/UTC" > /etc/timezone
+
+# Same mechanism GitHub's runner image uses to make system `pip install` work
+# on Ubuntu 24.04 despite PEP 668 (actions/runner-images install-python.sh):
+# a global pip.conf, not deleting the EXTERNALLY-MANAGED marker. Non-root pip
+# installs fall through to ~/.local, hence the PATH addition.
+RUN printf '[global]\nbreak-system-packages = true\n' > /etc/pip.conf
+
+# JDKs for the java-sdk prek hooks (sync-java-sdk-supervisor-schema, ktlint,
+# spotless run gradle on the host; toolchains reference Java 11 and 17).
+# GitHub-hosted runners ship the Temurin set — mirror the versions we use.
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends openjdk-11-jdk-headless openjdk-17-jdk-headless \
+ && rm -rf /var/lib/apt/lists/* \
+ && ln -s "/usr/lib/jvm/java-17-openjdk-$(dpkg --print-architecture)" /usr/lib/jvm/java-17
+ENV JAVA_HOME=/usr/lib/jvm/java-17
+
+# gh CLI (job-summary and finalize steps use `gh api`/`gh run view`; present
+# on GitHub-hosted runners). Official apt repo per cli.github.com docs.
+RUN curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg -o /etc/apt/keyrings/githubcli-archive-keyring.gpg \
+ && chmod a+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \
+ && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" > /etc/apt/sources.list.d/github-cli.list \
+ && apt-get update \
+ && apt-get install -y --no-install-recommends gh \
+ && rm -rf /var/lib/apt/lists/*
+
+# kubectl (hosted runners ship it; the k8s workflows' cluster-state dump steps
+# call it directly on the host). Pinned to the cluster's minor via pkgs.k8s.io.
+RUN curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key -o /tmp/k8s.key \
+ && gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg /tmp/k8s.key \
+ && chmod a+r /etc/apt/keyrings/kubernetes-apt-keyring.gpg \
+ && echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /" > /etc/apt/sources.list.d/kubernetes.list \
+ && apt-get update \
+ && apt-get install -y --no-install-recommends kubectl \
+ && rm -rf /var/lib/apt/lists/* /tmp/k8s.key
+
+# Real python3.10 on PATH (uv-managed build; hosted runners ship toolcache
+# pythons, so this matches the contract). Provider-compat jobs check out old
+# airflow branches whose prek configs pin python3.10 hooks; without a local
+# interpreter prek downloads its own managed CPython, and that download is
+# broken on arm64 (sysconfigdata missing build_time_vars, first ARM run
+# 2026-07-31). A PATH python short-circuits the download on both arches.
+RUN pip install uv \
+ && UV_PYTHON_INSTALL_DIR=/opt/uv-python uv python install 3.10 \
+ && ln -s /opt/uv-python/cpython-3.10.*/bin/python3.10 /usr/local/bin/python3.10 \
+ && /usr/local/bin/python3.10 -c "import sysconfig; sysconfig.get_path('stdlib')" \
+ && pip uninstall -y uv
+
+# AWS CLI v2 (stash actions use `aws s3`); official installer, arch-aware.
+RUN ARCH=$(uname -m) \
+ && curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-${ARCH}.zip" -o /tmp/awscliv2.zip \
+ && unzip -q /tmp/awscliv2.zip -d /tmp \
+ && /tmp/aws/install \
+ && rm -rf /tmp/aws /tmp/awscliv2.zip
+
+ENV PATH="/home/runner/.local/bin:${PATH}"
+
+# Hosted runners export USER; workflows rely on it (e.g. `sudo chown -R
+# "${USER}"` — with USER unset that chown is a silent no-op, exit 0, leaving
+# root-owned dirs that later non-sudo steps can't write).
+ENV USER=runner
+
+# breeze saves/loads CI image tars via `--image-file-dir /mnt` — on GitHub-hosted
+# runners /mnt is the big scratch disk made writable by make_mnt_writeable.sh,
+# which self-hosted runs deliberately skip. Backed here by the node's 200 GB
+# root volume; sticky like /tmp.
+RUN mkdir -p /mnt && chmod 1777 /mnt
+
+USER runner
diff --git a/terraform/README.rst b/terraform/README.rst
index ad8e00b..e9832c3 100644
--- a/terraform/README.rst
+++ b/terraform/README.rst
@@ -28,23 +28,37 @@ Requirements
In order to deploy the infrastructure, you need to have the following tools installed:
-- `tfenv <https://github.com/tfutils/tfenv>`_ to manage Terraform versions
+- Terraform >= 1.9 (e.g. via `tfenv <https://github.com/tfutils/tfenv>`_: ``tfenv install 1.9.5``)
- Configure your AWS and EKS credentials :ref:`doc <aws_configuration>`
+State
+~~~~~
-Configure Terraform
-~~~~~~~~~~~~~~~~~~~
+The remote state lives in the ``airflow-ci-tfstate`` S3 bucket (``us-east-2``) with the
+``TerraformLocks`` DynamoDB table. Credentials/profile are taken from the environment
+(``AWS_PROFILE=...``), which also works for a future GitHub Actions plan/apply workflow
+assuming a role via OIDC — nothing is hardcoded in the provider.
-.. code-block:: bash
+.. note::
- cd eks
- tfenv install 1.5.3
+ The bucket also contains ``tf-state/eks.tfstate`` — the **2024 deployment's** state.
+ Its cluster is gone but leftovers (the ``airflow-vpc`` VPC, old node-group IAM roles,
+ the ``/aws/eks/airflow/cluster`` log group, the ``alias/eks/airflow`` KMS alias)
+ still exist in the account and are **deliberately left untouched**. The current
+ deployment therefore uses its own state key (``tf-state/eks-airflow-ci.tfstate``)
+ and the cluster name ``airflow-ci``, so every derived resource name avoids the 2024
+ ones. Do not import, modify, or delete the 2024 resources from this configuration.
+ For local iteration without touching the shared state, drop a
+ ``backend_override.tf`` (gitignored) with ``terraform { backend "local" {} }``
+ next to the sources.
Plan and deploy the infrastructure
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
+ cd eks
+
# Initialize the Terraform modules and install the required providers,
# just the first time or when you add a new module
AWS_PROFILE=airflow terraform init
diff --git a/terraform/eks/.terraform.lock.hcl b/terraform/eks/.terraform.lock.hcl
index a3a0c54..b5d5d74 100644
--- a/terraform/eks/.terraform.lock.hcl
+++ b/terraform/eks/.terraform.lock.hcl
@@ -2,104 +2,109 @@
# Manual edits may be lost in future updates.
provider "registry.terraform.io/hashicorp/aws" {
- version = "5.64.0"
- constraints = ">= 4.33.0, >= 5.46.0, >= 5.61.0, ~> 5.64.0"
+ version = "6.56.0"
+ constraints = ">= 6.0.0, ~> 6.0, >= 6.28.0, >= 6.52.0"
hashes = [
- "h1:YH4I78rsS9t+YoGMPNzrM53aWi0Rb9Nud16iusrSXMg=",
- "zh:1d361f8062c68c9d5ac14b0aa8390709542129b8a9b258e61bbbabc706078b44",
- "zh:39dcbf53e3896bdd77071384c8fad4a5862c222c73f3bcf356aca488101f22fd",
- "zh:3fad63505f0c5b6f01cc9a6ef02b2226983b79424126a9caf6eb724f654299f4",
- "zh:53a8b90d00829cc27e3171a13a8ff1404ee0ea018e73f31d3f916d246cc39613",
- "zh:5734c25ef5a04b40f3c1ac5f817f11e42ee3328f74dbc141c0e64afbb0acc834",
- "zh:66ea14dbd87f291ce4a877123363933d3ca4022f209f885807a6689c22c24e80",
- "zh:68e79654ad0894a3d93134c3377748ace3058d5fad5ec09d1e9a8f8f9b8a47ea",
- "zh:7b74259d0ceef0c49cea6bcd171df997b6bad141085bbadded15b440faeb0eee",
- "zh:988ebfb5d115dc57070b5abf2e4200ad49cde535f27fd2ba5e34cf9ab336a57f",
+ "h1:iWz9BgFQaDPA1ChVGYvgI3MlUnx3wSQCA2ggYqDPcz8=",
+ "zh:2b3fbb3bebcc663b85d5fd9bbc2d131ab89322d696ff5c6ac6b7ffb7b5fe92e7",
+ "zh:30e56ccc7f33a7778ab323a28fe893d8e9200dc5fb92ccb7023bee808db3c1b0",
+ "zh:67dca271bef16547ef8ab5a6349f9bce39d91d7c1ae3d8388ada687ca774ba44",
+ "zh:824c812695b14d2fddad5e22339d4520e16d9c875f5d5095f29003f49a6fd124",
+ "zh:8372b12e30078d1df8b52e1285fd0d9d35160a7d18c3b0211f55229e5a832fd3",
+ "zh:8922b45ab65e272e8951f70d890444ddb3441170d8fa6f51298f24f20d21943d",
+ "zh:8fc4fb94ac8547717128cbcf296ac0ba0918738c4882029cbba46bd4812eb5e4",
"zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425",
- "zh:a0a2d4efe2835f0101a0a5024e044a3f28c00e10a8d87fce89c707ef6db75cea",
- "zh:aecb3e4b9121771dee9cac7975bf5d0657b5f3e8b57788c455beaeb0f3c48d93",
- "zh:d2d3393170b8ef761d3146f39f6788c4a3e876e6c5d4cedca4870c2680688ae6",
- "zh:daba5a005c1baa4a5eefbfb86d43ccf880eb5b42e8136f0d932f55886d72bda0",
- "zh:de16a6ff3baacdaf9609a0a89aa1913fc19cccaf5ee0fc1c49c5a075baa47c02",
+ "zh:a2efcd0174b79c1dffce48a5984f137ebc6f67d2c2ee966b47210e1faeb05cf2",
+ "zh:a4a50aa269a19c1d664b0dd59ee8047ed8a4a5b449f54733518309feccce1f43",
+ "zh:a830d80316b3c3127c9e0c77b9d4f50702c9d1a884c08973ac50b326d9ac734a",
+ "zh:d7e1b9bb2df3cdde8381ab101a807ae54e72c156d13a6b73dae2b922dca0c9f5",
+ "zh:d87c2a1cfc03b01cf13dff3700898ab990e1817326728824da8499dd0129da72",
+ "zh:fbb1848a597263c66dc6587ec8c3e0586e505e36f99d23752763534f62a3fef7",
+ "zh:fcb54d08eb3c763f01223e12b4eacbd018478e8e67c6b8611940564dfbea3d90",
+ "zh:ff6df70b2b2f75bd9000630f131c02e6bc2b9172cdb2547030f3fc019a3f3bc5",
]
}
provider "registry.terraform.io/hashicorp/cloudinit" {
- version = "2.3.4"
+ version = "2.4.0"
constraints = ">= 2.0.0"
hashes = [
- "h1:S3j8poSaLbaftlKq2STBkQEkZH253ZLaHhBHBifdpBQ=",
- "zh:09f1f1e1d232da96fbf9513b0fb5263bc2fe9bee85697aa15d40bb93835efbeb",
- "zh:381e74b90d7a038c3a8dcdcc2ce8c72d6b86da9f208a27f4b98cabe1a1032773",
- "zh:398eb321949e28c4c5f7c52e9b1f922a10d0b2b073b7db04cb69318d24ffc5a9",
- "zh:4a425679614a8f0fe440845828794e609b35af17db59134c4f9e56d61e979813",
- "zh:4d955d8608ece4984c9f1dacda2a59fdb4ea6b0243872f049b388181aab8c80a",
+ "h1:Bx3XQkBSY3RAGwLZb8hyi8AhvahPNlt4mlyZhW9guOI=",
+ "zh:1b0fe71b8e87a068f7cd9faaa733100ab72ab61ce812b8bd2b8e3e6ea3907b2d",
+ "zh:2aa9631ad64cfda1eb58f147619b631dadedfaf9453b422aa5ada2d3861183c1",
+ "zh:2c5f35463bdfb2f87d3576b81e62c30f8109e67bb6f21ffcbc46a855811455c0",
+ "zh:5970bcad151ea236bd262ada1a5a23bfbc1716f94a4e8b16ab2bcdda91d6a671",
"zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
- "zh:a48fbee1d58d55a1f4c92c2f38c83a37c8b2f2701ed1a3c926cefb0801fa446a",
- "zh:b748fe6631b16a1dafd35a09377c3bffa89552af584cf95f47568b6cd31fc241",
- "zh:d4b931f7a54603fa4692a2ec6e498b95464babd2be072bed5c7c2e140a280d99",
- "zh:f1c9337fcfe3a7be39d179eb7986c22a979cfb2c587c05f1b3b83064f41785c5",
- "zh:f58fc57edd1ee3250a28943cd84de3e4b744cdb52df0356a53403fc240240636",
- "zh:f5f50de0923ff530b03e1bca0ac697534d61bb3e5fc7f60e13becb62229097a9",
+ "zh:79a0676909732b6ec0441a733af6383513cde3bd2cef5c1ad0a74131e1286a04",
+ "zh:818f16141481a1202b3977becd19a12d4d46cd2e3f5753f5d0d0049adacf8f8c",
+ "zh:948d98716831087e69eca99f91ed7964cc537f3aca279f7494645ee56c9dc4ec",
+ "zh:a75e78889565a51df3e8e3af207e36e5ddb25e47ce1780a784c82dc3c3109b67",
+ "zh:a9c6e455d52b1bba5272bd87a35cfabcfd6d903dcbe42e2de926228dbb1e39b2",
+ "zh:b846805d8c2f5d1d6c2ffeeaf32109d9af7db7fa3c56929bfc1dcfaadf9c8bd8",
+ "zh:c3e5279756b46c4f49a6f4c81347fbe2fffebb2bf18a5c24664830304a1f6a8e",
+ "zh:c8be7b31893163d0046b0137a6100533f07e8efd192a1903b6bb4c42be12dceb",
]
}
provider "registry.terraform.io/hashicorp/null" {
- version = "3.2.2"
+ version = "3.3.0"
constraints = ">= 3.0.0"
hashes = [
- "h1:IMVAUHKoydFrlPrl9OzasDnw/8ntZFerCC9iXw1rXQY=",
- "zh:3248aae6a2198f3ec8394218d05bd5e42be59f43a3a7c0b71c66ec0df08b69e7",
- "zh:32b1aaa1c3013d33c245493f4a65465eab9436b454d250102729321a44c8ab9a",
- "zh:38eff7e470acb48f66380a73a5c7cdd76cc9b9c9ba9a7249c7991488abe22fe3",
- "zh:4c2f1faee67af104f5f9e711c4574ff4d298afaa8a420680b0cb55d7bbc65606",
- "zh:544b33b757c0b954dbb87db83a5ad921edd61f02f1dc86c6186a5ea86465b546",
- "zh:696cf785090e1e8cf1587499516b0494f47413b43cb99877ad97f5d0de3dc539",
- "zh:6e301f34757b5d265ae44467d95306d61bef5e41930be1365f5a8dcf80f59452",
+ "h1:l+dm3lhmu4ys7GbvIldfn544olSPH0DOiYruuFSfQkY=",
+ "zh:021748b5ea3b5f6956f2e75c42c5cdc113b391fb98ac71364a4965d23b37000f",
+ "zh:3b27956f8541d46704fda234e0d535c2ae2a4b33411848b1ee262a1ec03568b0",
+ "zh:3de4ed47d6d0f4d8edba4a5092c7c9799950eda63989d8d0d2586e6afcb0aa20",
+ "zh:57ed8935c7d56dbc91cf2673534582cacfaab7a2f105f51d9f797e99df0c0c47",
+ "zh:58e176ba1d142827089e30e0711e007309a9f2726e8881986da5026e9778fdf4",
+ "zh:5949c4a3d4a93f841f155cdb7e991c087e637145c1630572e21948224f8f4923",
+ "zh:76d60f366b743003c1b085afa769b45b2198ee919927e45807d7d44fb42c067d",
"zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
- "zh:913a929070c819e59e94bb37a2a253c228f83921136ff4a7aa1a178c7cce5422",
- "zh:aa9015926cd152425dbf86d1abdbc74bfe0e1ba3d26b3db35051d7b9ca9f72ae",
- "zh:bb04798b016e1e1d49bcc76d62c53b56c88c63d6f2dfe38821afef17c416a0e1",
- "zh:c23084e1b23577de22603cff752e59128d83cfecc2e6819edadd8cf7a10af11e",
+ "zh:79cd1bab1261a07f84e917191d7ddc4340ac5f5524283767256f7ffd7f87caf0",
+ "zh:8ec9083038cf710b30e319eaa467c9df7fa52bbd9969b61053a35bc2cdd2e0a6",
+ "zh:a6e502cb579685ab7aeb886c2bb11ddd9cfed74b41008592d57cbc3351a9218b",
+ "zh:acb74d6b4f66ff6acfcda315df802a7432170ef3955c9b432cb4580767004006",
+ "zh:f0ce55d8d9ffdb33dab612b1246f9bab060a9d54fc32ce2b4a038646155660af",
]
}
provider "registry.terraform.io/hashicorp/time" {
- version = "0.12.0"
+ version = "0.14.0"
constraints = ">= 0.9.0"
hashes = [
- "h1:Os2Ok7txtlUJHh6Hg7o+74Ql85SnRb/fGmah22yXpLw=",
- "zh:019a4c09af254ef80b72cf0d843dfe72d99483e227138cf5b514a1b9977ab4c3",
- "zh:0ae310ec740ebc6f275529507d60bb747d0bf39e72fc5a2fa90d74486006132c",
- "zh:13d6aec117f05237fbf8c7d91d6ebb19797b00aa87e7a812642d3ea4738a394e",
- "zh:2e87abbc261f9317d0c2ef26e01d5fabf77679da7d2cac6f47df7d198f720989",
- "zh:4a6d471176ce0264455aa7d5457b8702f78400010c201c1719708958a1b7b647",
+ "h1:4EThC3ocCFiFPMZQSUvSGSxoJqBcGWxMcFYmL67uS7Y=",
+ "zh:12abfd6b800e4d7fa6db7310dec8ffd440b31993861ef188c7ed5260b3073937",
+ "zh:23005521e800bb19e1597bf755c5f70d675d30b685d4255001ed5fa47d9df3f1",
+ "zh:2fea249b582ae97cd1cc10385187ea50993bb47c28cc5df0305e57ceaabf0a10",
+ "zh:322018d3b987b7aad08697178029a2bb667bed699e88328f0c89c52a2fd41341",
+ "zh:32a08e98fce2d273cb9b2c89d6c54727cc9f0a32e15bfd896be4e02cc6b48f95",
+ "zh:3db89aabd0e619616bd4b0f8b373a7586dfe60feffcea12a84a0bdbc445714b3",
+ "zh:7488f56c81d742dc020f29063626c8f07ca188aa97be61e7307e8d62397020a2",
"zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
- "zh:8978d5474a6da30bc0ad21c17db188d6918cacf3df3f6506b72ef3a268d53e2e",
- "zh:b109efe138dfcb45dc04a9cc6809d185ab8b0ebc12040847c2dac430fda5af68",
- "zh:b58e039b9106ac0a8de3c07f53b5279d7f0215fb35f2d23df642dfce0875382f",
- "zh:ba2cbb2e515922d13efe3a46647be84f5426fcfcaa0f1520b3efeab8db847ed3",
- "zh:c6c1ef1f26f25bca3abb5e07fa33dca37ed39cc26d0ff877964f2ffe5edd618c",
- "zh:f8e171f923b7d2e789abd034072465dec3e6133c3a7644b7a7a965a74d52224e",
+ "zh:7cb4067f2e7559b13f7562ef722f948950901eb37834873e98360ab28f66e9d7",
+ "zh:9d552c8345f61e1b7db8e725144981345f18ac1014d58d6f5ddf0928a195fffb",
+ "zh:a8e69fb6b97fc9d86fb19a9f4d42abe33c4a68e700b15387ce2e17d2b9934bed",
+ "zh:aeeb900eb8dd0f790c60ea5c0e0c8d42bd6e4a54f391681d4decca15b544394b",
+ "zh:c239c619101a8c95e1f14061eb973c57a8d15fa0e68878ced5bbd76858ee5b79",
]
}
provider "registry.terraform.io/hashicorp/tls" {
- version = "4.0.5"
- constraints = ">= 3.0.0"
+ version = "4.3.0"
+ constraints = ">= 4.0.0"
hashes = [
- "h1:zeG5RmggBZW/8JWIVrdaeSJa0OG62uFX5HY1eE8SjzY=",
- "zh:01cfb11cb74654c003f6d4e32bbef8f5969ee2856394a96d127da4949c65153e",
- "zh:0472ea1574026aa1e8ca82bb6df2c40cd0478e9336b7a8a64e652119a2fa4f32",
- "zh:1a8ddba2b1550c5d02003ea5d6cdda2eef6870ece86c5619f33edd699c9dc14b",
- "zh:1e3bb505c000adb12cdf60af5b08f0ed68bc3955b0d4d4a126db5ca4d429eb4a",
- "zh:6636401b2463c25e03e68a6b786acf91a311c78444b1dc4f97c539f9f78de22a",
- "zh:76858f9d8b460e7b2a338c477671d07286b0d287fd2d2e3214030ae8f61dd56e",
- "zh:a13b69fb43cb8746793b3069c4d897bb18f454290b496f19d03c3387d1c9a2dc",
- "zh:a90ca81bb9bb509063b736842250ecff0f886a91baae8de65c8430168001dad9",
- "zh:c4de401395936e41234f1956ebadbd2ed9f414e6908f27d578614aaa529870d4",
- "zh:c657e121af8fde19964482997f0de2d5173217274f6997e16389e7707ed8ece8",
- "zh:d68b07a67fbd604c38ec9733069fbf23441436fecf554de6c75c032f82e1ef19",
+ "h1:j/BqLS2N2AScZyotd9nZpHdieJ7e5S8y+A+ZfIu8kL8=",
+ "zh:0ab58d6f8991d436c7d2dbd89ed814709b949b07ac5a54ee53b0aec1fa772a8b",
+ "zh:60b347abcb56f45d97c56f14d895069cd15a83993f199777f571b79fea3642ee",
+ "zh:6889be32640349230de3f23856e6f04e0e9ced4a84a27d3f552fa54684448218",
+ "zh:73f8e1ecf7135033165fb14b7e8bf4d656f3ce13065ec35762ea0481975328c7",
+ "zh:94ce25ee253eca0b42cae9c856b36bca8103b6453012d1b279c3623c805f2d42",
+ "zh:96bc6de9fd67bc446fd11257872e1ffb1029a996ed1d65a3f6b43f6d408ad9ab",
+ "zh:97c609a310a51bfd504d704e036d72064a84bf0bdb36cc08cd4cc66098212b41",
+ "zh:a12c16e94533c5bd123f75032576b9dc91dd5d5ccd5f7cf331d0f2e1adc55cf8",
+ "zh:c4f014f876adf7af57188795050bda5b0029d8c7d7773031102b6c36dcf1fc21",
+ "zh:d9b0a21583aaa3df3a95394fb949a3c515ff71c2ff5a1fc4a73d364aa90bfca5",
+ "zh:da510d22f0c6d71ad19a76406f106b782448f512375787ecfabb338ed1e311a7",
+ "zh:f0e9447a9ce3a24cdaa113089e65663c836d8b9bfdb915a1c0284e0112cab5c0",
"zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
]
}
diff --git a/terraform/eks/asg_tags.tf b/terraform/eks/asg_tags.tf
deleted file mode 100644
index 3c05429..0000000
--- a/terraform/eks/asg_tags.tf
+++ /dev/null
@@ -1,67 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-######################################################################################################
-# In order to scale the node groups from zero, we need to add the labels and taints to the ASGs tags:
-# k8s.io/cluster-autoscaler/node-template/label/<label-key> = <label-value>
-# k8s.io/cluster-autoscaler/node-template/taint/<taint-key> = <taint-value>:<taint-effect>
-######################################################################################################
-
-locals {
-
- taint_effects = {
- NO_SCHEDULE = "NoSchedule"
- NO_EXECUTE = "NoExecute"
- PREFER_NO_SCHEDULE = "PreferNoSchedule"
- }
-
- cluster_autoscaler_label_tags = merge([
- for name, group in module.eks.eks_managed_node_groups : {
- for label_name, label_value in coalesce(group.node_group_labels, {}) : "${name}|label|${label_name}" => {
- autoscaling_group = group.node_group_autoscaling_group_names[0],
- key = "k8s.io/cluster-autoscaler/node-template/label/${label_name}",
- value = label_value,
- }
- }
- ]...)
-
- cluster_autoscaler_taint_tags = merge([
- for name, group in module.eks.eks_managed_node_groups : {
- for taint in coalesce(group.node_group_taints, []) : "${name}|taint|${taint.key}" => {
- autoscaling_group = group.node_group_autoscaling_group_names[0],
- key = "k8s.io/cluster-autoscaler/node-template/taint/${taint.key}"
- value = "${taint.value}:${local.taint_effects[taint.effect]}"
- }
- }
- ]...)
-
- cluster_autoscaler_asg_tags = merge(local.cluster_autoscaler_label_tags, local.cluster_autoscaler_taint_tags)
-}
-
-resource "aws_autoscaling_group_tag" "cluster_autoscaler_label_tags" {
- for_each = local.cluster_autoscaler_asg_tags
-
- autoscaling_group_name = each.value.autoscaling_group
-
- tag {
- key = each.value.key
- value = each.value.value
-
- propagate_at_launch = true
- }
-}
-
diff --git a/terraform/eks/eks.tf b/terraform/eks/eks.tf
index 5b4203a..bd901db 100644
--- a/terraform/eks/eks.tf
+++ b/terraform/eks/eks.tf
@@ -18,218 +18,59 @@
data "aws_availability_zones" "available" {}
locals {
- cluster_name = "airflow"
+ cluster_name = "airflow-ci"
}
module "eks" {
source = "terraform-aws-modules/eks/aws"
- version = "20.24.0"
+ version = "~> 21.24"
- cluster_name = local.cluster_name
- cluster_version = "1.30"
+ name = local.cluster_name
+ kubernetes_version = var.kubernetes_version
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.public_subnets
- cluster_endpoint_public_access = true
-
- eks_managed_node_group_defaults = {
- ami_type = "AL2_ARM_64"
- use_custom_launch_template = false
- disk_size = 50
- desired_size = 0
- }
-
- eks_managed_node_groups = {
-
- default_nodes = {
- name = "default"
-
- instance_types = [
- "t4g.medium",
- "t4g.large",
- "t4g.xlarge",
- "m7g.medium",
- "m7g.large",
- "m7g.xlarge",
- ]
-
- min_size = 0
- max_size = 5
- desired_size = 1
-
- capacity_type = "SPOT"
-
- labels = {
- "node-type" = "default"
- }
+ endpoint_public_access = true
+ enable_cluster_creator_admin_permissions = true
+
+ addons = {
+ coredns = {}
+ kube-proxy = {}
+ # before_compute: the module disables EKS's bootstrap self-managed CNI when
+ # managed addons are used, so vpc-cni must exist BEFORE nodes boot — nodes
+ # without a CNI never become Ready and the node group lands in CREATE_FAILED
+ # (bitten by this on 2026-07-25 when a crashed apply skipped the addons).
+ vpc-cni = {
+ before_compute = true
}
-
- # GHA arm runners' nodes
- GHA_runners_small = {
- name = "gha-runners-small"
-
- instance_types = [for node_type in var.runners_node_types: "${node_type}.${var.small_runners_node_size}"]
-
- min_size = 0
- max_size = 30
-
- capacity_type = "SPOT"
-
- disk_size = 50
-
- labels = {
- "node-type" = "gha-runners"
- "size" = "small"
- "arch" = "arm64"
- }
-
- taints = [
- {
- "key" = "node-type"
- "value" = "gha-runners"
- "effect" = "NO_SCHEDULE"
- }
- ]
+ # Required for Karpenter's Pod Identity association.
+ eks-pod-identity-agent = {
+ before_compute = true
}
+ }
- GHA_runners_medium = {
- name = "gha-runners-medium"
-
- instance_types = [for node_type in var.runners_node_types: "${node_type}.${var.medium_runners_node_size}"]
-
- min_size = 0
- max_size = 30
-
- capacity_type = "SPOT"
-
- disk_size = 50
-
- labels = {
- "node-type" = "gha-runners"
- "size" = "medium"
- "arch" = "arm64"
- }
-
- taints = [
- {
- "key" = "node-type"
- "value" = "gha-runners"
- "effect" = "NO_SCHEDULE"
- }
- ]
- }
-
- GHA_runners_large = {
- name = "gha-runners-large"
-
- instance_types = [for node_type in var.runners_node_types: "${node_type}.${var.large_runners_node_size}"]
-
- min_size = 0
- max_size = 30
-
- capacity_type = "SPOT"
-
- labels = {
- "node-type" = "gha-runners"
- "size" = "large"
- "arch" = "arm64"
- }
-
- taints = [
- {
- "key" = "node-type"
- "value" = "gha-runners"
- "effect" = "NO_SCHEDULE"
- }
- ]
- }
-
- # GHA x64 runners' nodes
- x64_GHA_runners_small = {
- name = "gha-x64-runners-small"
-
- ami_type = "AL2_x86_64"
-
- instance_types = [for node_type in var.x64_runners_node_types: "${node_type}.${var.small_runners_x64_node_size}"]
-
- min_size = 0
- max_size = 30
-
- capacity_type = "SPOT"
-
- disk_size = 50
-
- labels = {
- "node-type" = "gha-runners"
- "size" = "small"
- "arch" = "x64"
- }
-
- taints = [
- {
- "key" = "node-type"
- "value" = "gha-runners"
- "effect" = "NO_SCHEDULE"
- }
- ]
- }
-
- x64_GHA_runners_medium = {
- name = "gha-x64-runners-medium"
-
- ami_type = "AL2_x86_64"
-
- instance_types = [for node_type in var.x64_runners_node_types: "${node_type}.${var.medium_runners_node_size}"]
-
- min_size = 0
- max_size = 30
-
- capacity_type = "SPOT"
-
- disk_size = 50
-
- labels = {
- "node-type" = "gha-runners"
- "size" = "medium"
- "arch" = "x64"
- }
-
- taints = [
- {
- "key" = "node-type"
- "value" = "gha-runners"
- "effect" = "NO_SCHEDULE"
- }
- ]
- }
-
- x64_GHA_runners_large = {
- name = "gha-x64-runners-large"
-
- ami_type = "AL2_x86_64"
-
- instance_types = [for node_type in var.x64_runners_node_types: "${node_type}.${var.large_runners_node_size}"]
-
- min_size = 0
- max_size = 30
+ # Karpenter's EC2NodeClass discovers the node security group by this tag.
+ node_security_group_tags = {
+ "karpenter.sh/discovery" = local.cluster_name
+ }
- capacity_type = "SPOT"
+ eks_managed_node_groups = {
+ # Small always-on group for the controllers (Karpenter, ARC, listeners).
+ # Everything else runs on Karpenter-provisioned nodes.
+ system = {
+ name = "airflow-ci-system"
+ ami_type = "AL2023_ARM_64_STANDARD"
+ instance_types = var.system_node_instance_types
+ capacity_type = "ON_DEMAND"
+
+ min_size = 2
+ max_size = 3
+ desired_size = 2
labels = {
- "node-type" = "gha-runners"
- "size" = "large"
- "arch" = "x64"
+ "airflow-ci/role" = "system"
}
-
- taints = [
- {
- "key" = "node-type"
- "value" = "gha-runners"
- "effect" = "NO_SCHEDULE"
- }
- ]
}
}
}
-
diff --git a/terraform/eks/iam.tf b/terraform/eks/iam.tf
deleted file mode 100644
index e2c3000..0000000
--- a/terraform/eks/iam.tf
+++ /dev/null
@@ -1,134 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-data "aws_iam_policy_document" "autoscaler_policy_document" {
- statement {
- effect = "Allow"
- actions = [
- "autoscaling:DescribeAutoScalingGroups",
- "autoscaling:DescribeAutoScalingInstances",
- "autoscaling:DescribeLaunchConfigurations",
- "autoscaling:DescribeScalingActivities",
- "autoscaling:DescribeTags",
- "ec2:DescribeInstanceTypes",
- "ec2:DescribeLaunchTemplateVersions"
- ]
- resources = ["*"]
- }
-
- statement {
- effect = "Allow"
- actions = [
- "autoscaling:SetDesiredCapacity",
- "autoscaling:TerminateInstanceInAutoScalingGroup",
- "ec2:DescribeImages",
- "ec2:GetInstanceTypesFromInstanceRequirements",
- "eks:DescribeNodegroup"
- ]
- resources = ["*"]
- }
-}
-
-resource "aws_iam_policy" "autoscaler_policy" {
- name = "eks-autoscaler-policy"
- description = "EKS Autoscaler Policy"
- policy = data.aws_iam_policy_document.autoscaler_policy_document.json
-}
-
-data "aws_iam_policy_document" "autoscaler_role_assume_policy_document" {
- statement {
- actions = [
- "sts:AssumeRoleWithWebIdentity"
- ]
-
- principals {
- type = "Federated"
- identifiers = [
- module.eks.oidc_provider_arn
- ]
- }
-
- condition {
- test = "StringEquals"
- variable = "${module.eks.oidc_provider}:sub"
- values = [
- "system:serviceaccount:infra:cluster-autoscaler"
- ]
- }
-
- condition {
- test = "StringEquals"
- variable = "${module.eks.oidc_provider}:aud"
- values = [
- "sts.amazonaws.com"
- ]
- }
-
- }
-}
-
-resource "aws_iam_role" "autoscaler_role" {
- name = "eks-autoscaler-role"
- assume_role_policy = data.aws_iam_policy_document.autoscaler_role_assume_policy_document.json
-}
-
-resource "aws_iam_role_policy_attachment" "autoscaler_policy_attachment" {
- role = aws_iam_role.autoscaler_role.name
- policy_arn = aws_iam_policy.autoscaler_policy.arn
-}
-
-data "aws_iam_policy_document" "ebs_csi_driver_assume_policy_document" {
- statement {
- actions = [
- "sts:AssumeRoleWithWebIdentity"
- ]
-
- principals {
- type = "Federated"
- identifiers = [
- module.eks.oidc_provider_arn
- ]
- }
-
- condition {
- test = "StringEquals"
- variable = "${module.eks.oidc_provider}:sub"
- values = [
- "system:serviceaccount:kube-system:ebs-csi-controller-sa"
- ]
- }
-
- condition {
- test = "StringEquals"
- variable = "${module.eks.oidc_provider}:aud"
- values = [
- "sts.amazonaws.com"
- ]
- }
-
- }
-}
-
-resource "aws_iam_role" "ebs_csi_driver" {
- name = "ebs-csi-driver"
- assume_role_policy = data.aws_iam_policy_document.ebs_csi_driver_assume_policy_document.json
-}
-
-resource "aws_iam_role_policy_attachment" "ebs_csi_driver" {
- role = aws_iam_role.ebs_csi_driver.name
- policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
-}
diff --git a/terraform/eks/vpc.tf b/terraform/eks/karpenter.tf
similarity index 50%
copy from terraform/eks/vpc.tf
copy to terraform/eks/karpenter.tf
index 1d80025..c334d69 100644
--- a/terraform/eks/vpc.tf
+++ b/terraform/eks/karpenter.tf
@@ -15,30 +15,22 @@
# specific language governing permissions and limitations
# under the License.
-module "vpc" {
- source = "terraform-aws-modules/vpc/aws"
- version = "5.13.0"
+# IAM + SQS plumbing for Karpenter (controller role via EKS Pod Identity, node
+# role + access entry, spot-interruption queue). Karpenter itself is installed
+# with helmfile (helm/infra-helmfile.yaml); the names below are referenced from
+# the helm values, so they are pinned instead of name-prefixed.
+module "karpenter" {
+ source = "terraform-aws-modules/eks/aws//modules/karpenter"
+ version = "~> 21.24"
- name = "airflow-vpc"
+ cluster_name = module.eks.cluster_name
- cidr = "10.1.0.0/16"
- azs = slice(data.aws_availability_zones.available.names, 0, 3)
+ # Referenced by helm/values/karpenter.yaml (settings.interruptionQueue).
+ queue_name = "airflow-ci-karpenter"
- private_subnets = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"]
- public_subnets = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"]
+ # Referenced by the EC2NodeClass (spec.role) in helm/karpenter-resources.
+ node_iam_role_name = "airflow-ci-runner-node"
+ node_iam_role_use_name_prefix = false
- enable_nat_gateway = false
- enable_dns_hostnames = true
-
- map_public_ip_on_launch = true
-
- public_subnet_tags = {
- "kubernetes.io/cluster/${local.cluster_name}" = "shared"
- "kubernetes.io/role/elb" = 1
- }
-
- private_subnet_tags = {
- "kubernetes.io/cluster/${local.cluster_name}" = "shared"
- "kubernetes.io/role/internal-elb" = 1
- }
-}
\ No newline at end of file
+ create_pod_identity_association = true
+}
diff --git a/terraform/eks/vpc.tf b/terraform/eks/outputs.tf
similarity index 50%
copy from terraform/eks/vpc.tf
copy to terraform/eks/outputs.tf
index 1d80025..f7d224d 100644
--- a/terraform/eks/vpc.tf
+++ b/terraform/eks/outputs.tf
@@ -15,30 +15,27 @@
# specific language governing permissions and limitations
# under the License.
-module "vpc" {
- source = "terraform-aws-modules/vpc/aws"
- version = "5.13.0"
-
- name = "airflow-vpc"
-
- cidr = "10.1.0.0/16"
- azs = slice(data.aws_availability_zones.available.names, 0, 3)
-
- private_subnets = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"]
- public_subnets = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"]
-
- enable_nat_gateway = false
- enable_dns_hostnames = true
-
- map_public_ip_on_launch = true
-
- public_subnet_tags = {
- "kubernetes.io/cluster/${local.cluster_name}" = "shared"
- "kubernetes.io/role/elb" = 1
- }
-
- private_subnet_tags = {
- "kubernetes.io/cluster/${local.cluster_name}" = "shared"
- "kubernetes.io/role/internal-elb" = 1
- }
-}
\ No newline at end of file
+output "cluster_name" {
+ description = "EKS cluster name"
+ value = module.eks.cluster_name
+}
+
+output "cluster_endpoint" {
+ description = "EKS API endpoint"
+ value = module.eks.cluster_endpoint
+}
+
+output "karpenter_node_iam_role_name" {
+ description = "IAM role assumed by Karpenter-launched nodes (EC2NodeClass spec.role)"
+ value = module.karpenter.node_iam_role_name
+}
+
+output "karpenter_interruption_queue" {
+ description = "SQS queue for spot interruption handling (karpenter settings.interruptionQueue)"
+ value = module.karpenter.queue_name
+}
+
+output "update_kubeconfig" {
+ description = "Command to configure kubectl"
+ value = "aws eks --region ${var.region} update-kubeconfig --name ${module.eks.cluster_name} --alias ${module.eks.cluster_name}"
+}
diff --git a/terraform/eks/provider.tf b/terraform/eks/provider.tf
index 89656fe..5388c88 100644
--- a/terraform/eks/provider.tf
+++ b/terraform/eks/provider.tf
@@ -15,7 +15,16 @@
# specific language governing permissions and limitations
# under the License.
+# Credentials come from the environment (AWS_PROFILE locally, OIDC role in a
+# future GitHub Actions workflow) — do not hardcode a profile here.
provider "aws" {
- profile = "airflow"
- region = var.region
+ region = var.region
+
+ default_tags {
+ tags = {
+ Project = "airflow-ci"
+ ManagedBy = "terraform"
+ Repo = "apache/airflow-ci-infra"
+ }
+ }
}
\ No newline at end of file
diff --git a/terraform/eks/runner_pod_identity.tf b/terraform/eks/runner_pod_identity.tf
new file mode 100644
index 0000000..bbd267d
--- /dev/null
+++ b/terraform/eks/runner_pod_identity.tf
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The airflow CI workflows pass artifacts between jobs via the S3-backed stash
+# actions (.github/actions/stash-s3-*). Runner pods get scoped S3 access
+# through EKS Pod Identity — node IAM credentials are deliberately
+# unreachable from pods (IMDS hop limit 1 on the EC2NodeClass).
+locals {
+ # Same-region successor of the us-east-1 `airflow-ci-runner-cache` bucket
+ # (bucket names are global, so the regional suffix disambiguates). Contents
+ # are disposable inter-job artifacts — nothing was migrated.
+ stash_bucket = "airflow-ci-runner-cache-us-east-2"
+
+ # One service account per runner scale set, created by the gha-runner-scale-set
+ # chart as "<scale-set-name>-gha-rs-no-permission" in the arc-runners namespace.
+ runner_scale_set_names = [
+ "airflow-runner-amd-small",
+ "airflow-runner-amd-medium",
+ "airflow-runner-amd-xl",
+ "airflow-runner-arm-small",
+ "airflow-runner-arm-medium",
+ "airflow-runner-arm-xl",
+ ]
+}
+
+resource "aws_s3_bucket" "stash" {
+ bucket = local.stash_bucket
+}
+
+resource "aws_s3_bucket_public_access_block" "stash" {
+ bucket = aws_s3_bucket.stash.id
+
+ block_public_acls = true
+ block_public_policy = true
+ ignore_public_acls = true
+ restrict_public_buckets = true
+}
+
+# Stashes are short-lived hand-offs between jobs of a run; the actions ask for
+# ~2-day retention, the lifecycle rule is the backstop that actually deletes.
+resource "aws_s3_bucket_lifecycle_configuration" "stash" {
+ bucket = aws_s3_bucket.stash.id
+
+ rule {
+ id = "expire-stashes"
+ status = "Enabled"
+
+ filter {}
+
+ expiration {
+ days = 3
+ }
+
+ abort_incomplete_multipart_upload {
+ days_after_initiation = 1
+ }
+ }
+}
+
+data "aws_iam_policy_document" "runner_pods_assume" {
+ statement {
+ actions = ["sts:AssumeRole", "sts:TagSession"]
+ principals {
+ type = "Service"
+ identifiers = ["pods.eks.amazonaws.com"]
+ }
+ }
+}
+
+data "aws_iam_policy_document" "runner_pods_stash" {
+ statement {
+ actions = ["s3:GetObject", "s3:PutObject"]
+ resources = ["arn:aws:s3:::${local.stash_bucket}/*"]
+ }
+ statement {
+ actions = ["s3:ListBucket", "s3:GetBucketLocation"]
+ resources = ["arn:aws:s3:::${local.stash_bucket}"]
+ }
+}
+
+resource "aws_iam_role" "runner_pods" {
+ name = "airflow-ci-runner-pods"
+ assume_role_policy = data.aws_iam_policy_document.runner_pods_assume.json
+}
+
+resource "aws_iam_role_policy" "runner_pods_stash" {
+ name = "stash-bucket-access"
+ role = aws_iam_role.runner_pods.id
+ policy = data.aws_iam_policy_document.runner_pods_stash.json
+}
+
+resource "aws_eks_pod_identity_association" "runner_pods" {
+ for_each = toset(local.runner_scale_set_names)
+
+ cluster_name = module.eks.cluster_name
+ namespace = "arc-runners"
+ service_account = "${each.value}-gha-rs-no-permission"
+ role_arn = aws_iam_role.runner_pods.arn
+}
diff --git a/terraform/eks/state.tf b/terraform/eks/state.tf
index 025e912..8801ae2 100644
--- a/terraform/eks/state.tf
+++ b/terraform/eks/state.tf
@@ -18,10 +18,16 @@
terraform {
backend "s3" {
- region = "us-east-2"
- bucket = "airflow-ci-tfstate"
- key = "tf-state/eks.tfstate"
- encrypt = true
- dynamodb_table = "TerraformLocks"
+ region = "us-east-2"
+ bucket = "airflow-ci-tfstate"
+ # Fresh key for the 2026 airflow-ci cluster. tf-state/eks.tfstate is the
+ # 2024 deployment's state and is deliberately left untouched (its cluster
+ # is gone but the VPC/IAM leftovers still exist and must not be disturbed).
+ key = "tf-state/eks-airflow-ci.tfstate"
+ encrypt = true
+ # S3-native locking (.tflock object next to the state). Replaces the
+ # deprecated dynamodb_table locking; the TerraformLocks DynamoDB table is
+ # still referenced by the 2024 state and must not be deleted.
+ use_lockfile = true
}
}
diff --git a/terraform/eks/variables.tf b/terraform/eks/variables.tf
index 045c496..deeb5a0 100644
--- a/terraform/eks/variables.tf
+++ b/terraform/eks/variables.tf
@@ -21,48 +21,14 @@ variable "region" {
default = "us-east-2"
}
-variable "runners_node_types" {
- description = "Node type for the runners"
- type = list(string)
- default = [
- "t4g",
- "m7g",
- "m6g",
- ]
-}
-
-variable "x64_runners_node_types" {
- description = "X64 node type for the runners"
- type = list(string)
- default = [
- "m7a",
- "m6a",
- "t3",
- "t3a",
- ]
-}
-
-variable "small_runners_node_size" {
- description = "Node size for the small runners"
- type = string
- default = "medium"
-}
-
-variable "small_runners_x64_node_size" {
- # This is a separate variable because the r series doesn't have a medium size
- description = "Node size for the small runners"
+variable "kubernetes_version" {
+ description = "EKS Kubernetes version; keep within standard support to avoid extended-support fees"
type = string
- default = "large"
+ default = "1.34"
}
-variable "medium_runners_node_size" {
- description = "Node size for the medium runners"
- type = string
- default = "xlarge"
+variable "system_node_instance_types" {
+ description = "Instance types for the small always-on system node group (Karpenter + ARC controllers)"
+ type = list(string)
+ default = ["t4g.medium"]
}
-
-variable "large_runners_node_size" {
- description = "Node size for the large runners"
- type = string
- default = "2xlarge"
-}
\ No newline at end of file
diff --git a/terraform/eks/version.tf b/terraform/eks/version.tf
index a215415..93977fd 100644
--- a/terraform/eks/version.tf
+++ b/terraform/eks/version.tf
@@ -19,8 +19,9 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = "~> 5.64.0"
+ version = "~> 6.0"
}
}
- required_version = "~> 1.9.5"
+ # 1.10+ for S3-native state locking (use_lockfile)
+ required_version = ">= 1.10.0"
}
\ No newline at end of file
diff --git a/terraform/eks/vpc.tf b/terraform/eks/vpc.tf
index 1d80025..3ff2339 100644
--- a/terraform/eks/vpc.tf
+++ b/terraform/eks/vpc.tf
@@ -15,17 +15,21 @@
# specific language governing permissions and limitations
# under the License.
+# Nodes live in public subnets with public IPs on purpose: a NAT gateway costs
+# ~$400/yr while public IPv4 on ephemeral runner nodes is a few $/mo, and the
+# security groups allow no inbound traffic (runners are outbound-only).
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
- version = "5.13.0"
+ version = "~> 6.6"
- name = "airflow-vpc"
+ name = "airflow-ci-vpc"
cidr = "10.1.0.0/16"
azs = slice(data.aws_availability_zones.available.names, 0, 3)
- private_subnets = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"]
- public_subnets = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"]
+ # Docker inside the runners carves networks out of 172.16.0.0/14 — the VPC
+ # range must stay clear of it, hence 10.x.
+ public_subnets = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"]
enable_nat_gateway = false
enable_dns_hostnames = true
@@ -34,11 +38,7 @@ module "vpc" {
public_subnet_tags = {
"kubernetes.io/cluster/${local.cluster_name}" = "shared"
- "kubernetes.io/role/elb" = 1
+ # Karpenter's EC2NodeClass discovers subnets by this tag.
+ "karpenter.sh/discovery" = local.cluster_name
}
-
- private_subnet_tags = {
- "kubernetes.io/cluster/${local.cluster_name}" = "shared"
- "kubernetes.io/role/internal-elb" = 1
- }
-}
\ No newline at end of file
+}