You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by sh...@apache.org on 2026/09/17 10:01:59 UTC
(airflow-ci-infra) branch gha-arc-rebase updated (aec1067 -> 4dbb262)
This is an automated email from the ASF dual-hosted git repository.
shahar1 pushed a change to branch gha-arc-rebase
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git
discard aec1067 Replace static node groups with Karpenter and ARC runner scale sets
new 4dbb262 Replace static node groups with Karpenter and ARC runner scale sets
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (aec1067)
\
N -- N -- N refs/heads/gha-arc-rebase (4dbb262)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/eks-runner-architecture.md | 24 +++++++++++-----------
helm/karpenter-resources/values.yaml | 2 +-
.../gha-runner-scale-sets/common.yaml.gotmpl | 2 +-
3 files changed, 14 insertions(+), 14 deletions(-)
(airflow-ci-infra) 01/01: Replace static node groups with Karpenter and ARC runner scale sets
Posted by sh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
shahar1 pushed a commit to branch gha-arc-rebase
in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git
commit 4dbb262a11abe8bff30b93b42d77d3ee84e4a380
Author: Shahar Epstein <60...@users.noreply.github.com>
AuthorDate: Thu Sep 17 13:00:36 2026 +0300
Replace static node groups with Karpenter and ARC runner scale sets
The previous design (cluster-autoscaler, six managed node groups, legacy
ARC with cert-manager) could not scale from zero without ASG tag
workarounds and had no spot-price-aware instance selection. An earlier
managed-runner experiment measured ~$104 per full Tests (AMD) run against
a ~$15k/yr CI budget. Karpenter with EKS Pod Identity and ARC's supported
scale-set mode brings the same run to ~$12-13 with one NodePool per
architecture and no cert-manager or EBS CSI dependency.
The design rationale lives in docs/eks-runner-architecture.md and ADR 0001;
the runner image now matches the host contract airflow's CI graph expects.
---
Airflow_CI.png | Bin 80422 -> 0 bytes
CONTEXT.md | 47 ++++
README.rst | 122 +++++------
...ks-standard-mode-with-self-managed-karpenter.md | 40 ++++
docs/eks-runner-architecture.md | 244 +++++++++++++++++++++
helm/{ci-helmfile.yaml => ci-helmfile.yaml.gotmpl} | 15 +-
.../templates/namespaces/{infra.yaml => arc.yaml} | 9 +-
helm/cluster-config/templates/namespaces/ci.yaml | 26 ---
helm/infra-helmfile.yaml | 42 ++--
.../Chart.yaml} | 9 +-
.../templates/ec2nodeclass.yaml | 71 ++++++
.../templates/nodepool-amd-shared.yaml | 65 ++++++
.../templates/nodepool-amd.yaml | 63 ++++++
.../templates/nodepool-arm-shared.yaml | 65 ++++++
.../templates/nodepool-arm.yaml | 63 ++++++
.../values.yaml} | 30 +--
helm/values/cert-manager.yaml | 20 --
helm/values/gha-runner-scale-set-controller.yaml | 5 +-
.../gha-runner-scale-sets/common.yaml.gotmpl | 137 ++++++++++--
helm/values/gha-runner-scale-sets/runners.yaml | 97 +++++---
.../{cluster-autoscaler.yaml => karpenter.yaml} | 28 +--
runner/Dockerfile | 88 +++++++-
terraform/README.rst | 26 ++-
terraform/eks/.terraform.lock.hcl | 145 ++++++------
terraform/eks/asg_tags.tf | 67 ------
terraform/eks/eks.tf | 233 ++++----------------
terraform/eks/iam.tf | 134 -----------
terraform/eks/{vpc.tf => karpenter.tf} | 38 ++--
terraform/eks/{vpc.tf => outputs.tf} | 51 ++---
terraform/eks/provider.tf | 13 +-
terraform/eks/runner_pod_identity.tf | 113 ++++++++++
terraform/eks/state.tf | 16 +-
terraform/eks/variables.tf | 48 +---
terraform/eks/version.tf | 5 +-
terraform/eks/vpc.tf | 22 +-
35 files changed, 1376 insertions(+), 821 deletions(-)
diff --git a/Airflow_CI.png b/Airflow_CI.png
deleted file mode 100644
index 82e5043..0000000
Binary files a/Airflow_CI.png and /dev/null differ
diff --git a/CONTEXT.md b/CONTEXT.md
new file mode 100644
index 0000000..901412a
--- /dev/null
+++ b/CONTEXT.md
@@ -0,0 +1,47 @@
+# Airflow CI Infrastructure
+
+Self-hosted GitHub Actions runner infrastructure (EKS + ARC + Karpenter) that offloads
+selected Apache Airflow CI workloads from GitHub-hosted runners to AWS.
+
+## Language
+
+### Workload categories
+
+**Trusted run**:
+A workflow run whose code ref is not attacker-controllable: `schedule`, `push`, or
+`workflow_dispatch` events on `apache/airflow` itself. The only category eligible for
+self-hosted runners in phase 1.
+_Avoid_: using "trusted" for committer-authored PRs — those are Committer runs.
+
+**Canary run**:
+A full-matrix CI run of main, triggered by cron (`schedule`) or by a push to main.
+"Scheduled canary" and "push canary" distinguish the trigger when it matters.
+
+**Committer run**:
+A CI run of a PR authored by someone with merge access (regardless of whether it comes
+from a fork or a branch). Not a Trusted run — the PR code is still unreviewed; serving
+these from self-hosted runners is a separate, unresolved decision.
+_Avoid_: trusted PR, member run.
+
+**Fork PR run**:
+A CI run of a PR from a contributor without merge access. Never eligible for
+self-hosted runners.
+
+### Infrastructure
+
+**Runner scale set**:
+An ARC unit of runner capacity targeted by a single `runs-on` label (e.g.
+`airflow-runner-amd-xl`). One helm release per scale set.
+_Avoid_: runner deployment (legacy ARC term), runner pool.
+
+**Runner class**:
+The size tier of a scale set, expressed as pod resource requests (CPU/memory), not as a
+node type. Karpenter chooses the actual instance.
+
+**System node group**:
+The small always-on EKS managed node group hosting the controllers (ARC, Karpenter,
+listeners). Everything else runs on Karpenter-provisioned nodes.
+
+**Stash**:
+The S3-backed mechanism for passing artifacts between jobs of a run (drop-in for the
+`apache/infrastructure-actions/stash` actions).
diff --git a/README.rst b/README.rst
index 729c657..f3c1e43 100644
--- a/README.rst
+++ b/README.rst
@@ -53,7 +53,7 @@ you need to follow the steps below:
3. Update kubeconfig to access the EKS cluster::
- AWS_PROFILE=airflow aws eks --region us-east-2 update-kubeconfig --name airflow --alias airflow
+ AWS_PROFILE=airflow aws eks --region us-east-2 update-kubeconfig --name airflow-ci --alias airflow-ci
This command will update the kubeconfig file located at `~/.kube/config` to include the EKS cluster.
You can verify that the cluster is added by running `kubectl get nodes`.
@@ -65,82 +65,82 @@ you need to follow the steps below:
Airflow CI infrastructure architecture
--------------------------------------
-.. image:: Airflow_CI.png
- :width: 100%
- :align: center
- :alt: Airflow CI infrastructure architecture
-Manage the infrastructure resources
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The design is documented in `docs/eks-runner-architecture.md <docs/eks-runner-architecture.md>`_.
+In short: an EKS cluster (standard mode) with a small always-on system node group runs
+`Actions Runner Controller <https://github.com/actions/actions-runner-controller>`_ (runner
+scale sets mode) and `Karpenter <https://karpenter.sh>`_; runner pods are ephemeral
+docker-in-docker pods, one job per pod, on spot nodes that Karpenter provisions on demand
+and reclaims when idle.
-We use Terraform to manage the EKS cluster and other AWS resources, and Helm (with helmfile) to manage the
-Kubernetes resources inside the cluster.
+Deploying from scratch
+~~~~~~~~~~~~~~~~~~~~~~
-For Terraform, you can find the detailed documentation here: `Terraform Documentation <terraform/README.rst>`_
+We use Terraform for the AWS resources and Helm (with helmfile) for everything inside the
+cluster. The order matters on a fresh account:
-And for Helm, you mainly need two commands:
+1. **Terraform** — see `Terraform Documentation <terraform/README.rst>`_::
-.. code-block:: bash
+ cd terraform/eks
+ terraform init
+ terraform plan -out=terraform.tfplan
+ terraform apply terraform.tfplan
- # diff the infra helmfile to compare the current state with the desired state
- helmfile -f helm/infra-helmfile.yaml diff
+2. **kubeconfig**::
- # apply the helmfiles to deploy the desired state
- helmfile -f helm/infra-helmfile.yaml apply
+ aws eks --region us-east-2 update-kubeconfig --name airflow-ci --alias airflow-ci
- # same for ci helmfile
- helmfile -f helm/ci-helmfile.yaml diff
- helmfile -f helm/ci-helmfile.yaml apply
+3. **Infra helmfile** (namespaces, Karpenter, NodePools)::
-For Docker, we build a custom image for the runners, based on the official `Dockerfiles`_ provided by Github, but
-with the `Airflow runner release`_ managed by `Ash`_ instead of the official one, and with python installed on the image
-(check `setup-python issue`_ for more details).
+ helmfile -f helm/infra-helmfile.yaml diff
+ helmfile -f helm/infra-helmfile.yaml apply
-To build docker image, you can run the following command:
+4. **GitHub App secret** — create a GitHub App (for repository-level runners:
+ ``Administration: Read & write`` + ``Metadata: Read-only``; for organization-level
+ runners: ``Self-hosted runners: Read & write``), install it on the target
+ repo/org, then::
-.. code-block:: bash
+ kubectl create secret generic github-config --namespace arc-runners \
+ --from-literal=github_app_id=<app-id> \
+ --from-literal=github_app_installation_id=<installation-id> \
+ --from-file=github_app_private_key=<path-to-private-key.pem>
- export TARGETPLATFORM=linux/arm64
- export AIRFLOW_RUNNER_VERSION=2.304.0-airflow8
- docker build --platform $TARGETPLATFORM \
- --build-arg AIRFLOW_RUNNER_VERSION=$AIRFLOW_RUNNER_VERSION \
- -t public.ecr.aws/u9s5q9f7/airflow-gha-runner:$AIRFLOW_RUNNER_VERSION \
- runner --push
+5. **CI helmfile** (ARC controller + runner scale sets)::
-.. _Dockerfiles: https://github.com/actions/actions-runner-controller/tree/master/runner
+ helmfile -f helm/ci-helmfile.yaml.gotmpl diff
+ helmfile -f helm/ci-helmfile.yaml.gotmpl apply
-.. _Airflow runner release: https://github.com/ashb/runner/tree/v2.304.0-airflow8
+The runner image extends the official ``ghcr.io/actions/actions-runner`` image with the
+host tools airflow's CI expects (sudo, docker CLI + compose, python3, ...). Images are
+**date-tagged, never ``:latest``** — the tag referenced in
+``helm/values/gha-runner-scale-sets/runners.yaml`` must always be an explicit, reviewed
+pin. Build and push::
-.. _Ash: https://github.com/ashb
-
-.. _setup-python issue: https://github.com/actions/setup-python/issues/705
+ TAG=$(date +%Y%m%d)
+ docker build --platform linux/amd64 \
+ -t ghcr.io/apache/airflow-ci-infra/actions-runner:${TAG} runner
+ docker push ghcr.io/apache/airflow-ci-infra/actions-runner:${TAG}
+ # then update the image tag in helm/values/gha-runner-scale-sets/runners.yaml
Some important notes
~~~~~~~~~~~~~~~~~~~~
-There are some configuration parameters that you need to be aware of:
-
-- EKS node group instance type:
- In our current architecture, we use different types of runners, and each set of runners is configured via a
- RunnerDeployment. Instead of setting container resources, we define the node selectors to schedule the
- runner pod on a specific node group, and we add an anti-affinity to ensure the pod uses all the node resources
- without sharing them with other runners. In this case, we need to carefully choose the instance types because they
- represent the runners' resources.
-
-- ARC sync period:
- The ARC sync period is the duration in which the controller reconciles the desired runners count. Increasing
- this period could lead to a delay in the runners count update (more time in the queue when needing a scale-up
- and a waste of resources when needing a scale-down). Decreasing it could lead to a high load on the Github API.
-
-- Autoscaler conf:
- Similar to the ARC sync period, the autoscaler `scan-interval` is the duration in which the autoscaler checks the
- cluster resources and scales up or down the cluster. Additionally, two other important parameters are
- `scale-down-unneeded-time` and `scale-down-delay-after-add`, which control the delay before taking the scale-down
- decision when the node is underutilized and the delay of checking the scaling-down necessity after a scale-up
- decision.
-
-- HorizontalRunnerAutoscaler scaleDownDelaySecondsAfterScaleOut:
- This parameter represents the delay before the HorizontalRunnerAutoscaler (HRA) takes the scale-down decision after
- adding a new runner. Currently, we use a small delay (30 seconds) because the K8S autoscaler waits for 5 minutes
- before deleting the node. So, if we need a runner, the HRA will add a new one, and it will be scheduled in less
- than 20 seconds.
+- Runner sizing is done with pod **resource requests** (see
+ ``helm/values/gha-runner-scale-sets/runners.yaml``), not node selectors/anti-affinity:
+ Karpenter picks the cheapest spot instance that fits the request. The current single
+ ``airflow-runner-amd-xl`` class requests a whole 32-vCPU node per job.
+- Scale-up path: job queued → ARC listener requests a runner pod → pod Pending →
+ Karpenter launches a node (~60-90s). Scale-down: pod finishes → node empty →
+ Karpenter consolidates it after 60s (``WhenEmpty`` — running jobs are never evicted).
+- The dockerd inside each runner is configured (dind sidecar args in
+ ``helm/values/gha-runner-scale-sets/common.yaml.gotmpl``) with a registry mirror,
+ widened address pools and nofile ulimits — these fix real failures observed with
+ airflow's CI and should not be removed casually.
+- Workflows target the runners with ``runs-on: <scale-set-name>``, e.g.
+ ``runs-on: airflow-runner-amd-xl``.
+- **Spot policy (phase 1): spot-first for everything, deliberately.** A spot reclaim
+ kills the job mid-run — the job shows red with a "runner lost communication"
+ annotation; just re-run the failed jobs. Tripwire: if interruptions exceed ~2/week
+ (or kill the same canary twice in a row), pin the heavy runner class to on-demand
+ (``karpenter.sh/capacity-type`` requirement in the NodePool) once size classes are
+ split.
diff --git a/docs/adr/0001-eks-standard-mode-with-self-managed-karpenter.md b/docs/adr/0001-eks-standard-mode-with-self-managed-karpenter.md
new file mode 100644
index 0000000..28f7b23
--- /dev/null
+++ b/docs/adr/0001-eks-standard-mode-with-self-managed-karpenter.md
@@ -0,0 +1,40 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# EKS standard mode with self-managed Karpenter, not EKS Auto Mode
+
+Status: accepted (2026-07-25)
+
+The CI runner cluster runs EKS in standard mode with Karpenter installed and
+managed by us, even though EKS Auto Mode is AWS's recommended default and would
+remove Karpenter, AMI, and addon management entirely. Auto Mode nodes are
+locked-down Bottlerocket appliances — SELinux enforcing, immutable read-only
+root, no SSH/SSM, no node-level customization — and our runner pods are
+privileged docker-in-docker workloads that run KinD (Kubernetes *inside* that
+dind) and require node sysctls (`fs.inotify.*`, `nf_conntrack`, `fs.nr_open`);
+this is precisely the workload class that fights a locked host, and it is
+battle-tested on plain AL2023 with our own user data. Auto Mode also adds a
+per-instance management fee (~10–12% of the instance price) to a
+compute-dominated, budget-capped bill.
+
+Revisit if the workload ever stops needing privileged dind (e.g. jobs move to
+rootless/kaniko-style builds and KinD is replaced), or if Auto Mode gains
+supported node-level sysctl/privileged-workload customization at a fee that
+undercuts our measured operational cost of self-managing Karpenter
+(~a few hours per quarter).
diff --git a/docs/eks-runner-architecture.md b/docs/eks-runner-architecture.md
new file mode 100644
index 0000000..597bc64
--- /dev/null
+++ b/docs/eks-runner-architecture.md
@@ -0,0 +1,244 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# EKS-based GitHub Actions runners for Airflow CI — architecture
+
+*Status: proposal (2026-07). Supersedes the 2023/2024 design on this branch (PR #59).
+All versions and prices below were checked against official sources on 2026-07-25.*
+
+## Why this exists
+
+Airflow's canary/scheduled CI competes for the shared ASF GitHub Actions concurrency
+pool. An earlier managed-runner experiment (June–July 2026) proved the mechanics
+work but the economics don't: a full "Tests (AMD)" run cost **≈$104**
+(1,300 build-minutes on 36 vCPU machines at $0.0798/min), ≈$135k+/yr for scheduled
+runs alone — far beyond the ~$15k/yr CI budget envelope. Measured per-job utilization
+shows a **7.3× oversizing waste** (median job peaks at ~3 vCPU / 2.3 GiB on a
+36 vCPU / 72 GiB box) on top of that service's ~6.6× per-vCPU price premium over EC2 spot.
+EKS + spot brings the same run to **≈$7–13**.
+
+## Architecture at a glance
+
+```mermaid
+flowchart LR
+ subgraph github["GitHub"]
+ actions["GitHub Actions service"]
+ workflow["apache/airflow workflows<br/>runs-on: airflow-runner-amd-xl"]
+ workflow -->|"queue job"| actions
+ end
+
+ subgraph aws["AWS account · us-east-2"]
+ subgraph vpc["airflow-ci-vpc — public subnets, no NAT"]
+ subgraph eks["EKS cluster 'airflow-ci' v1.34 (standard mode)"]
+ subgraph system["System node group — 2× t4g.medium on-demand"]
+ controller["ARC controller<br/>(arc-systems)"]
+ listener["ARC listener<br/>per scale set"]
+ karpenter["Karpenter v1.11<br/>(kube-system)"]
+ end
+ subgraph arcrunners["arc-runners namespace"]
+ secret["Secret github-config<br/>(GitHub App creds)"]
+ runner["Ephemeral runner pod<br/>runner + privileged dind sidecar<br/>requests 28 vCPU / 56 Gi · 1 job"]
+ end
+ node["Spot node — cheapest of<br/>c/m/r 6th-7th gen ≥ 8xlarge<br/>AL2023 · 200 Gi gp3 · IMDS hop 1"]
+ end
+ end
+ sqs["SQS queue<br/>airflow-ci-karpenter<br/>(spot interruptions)"]
+ s3[("S3 stash bucket<br/>airflow-ci-runner-cache-us-east-2")]
+ end
+
+ listener -->|"outbound long-poll<br/>(no inbound access)"| actions
+ listener -->|"job queued:<br/>request runner"| runner
+ secret -.-> listener
+ runner -.->|"Pending pod"| karpenter
+ karpenter -->|"launch / consolidate<br/>(WhenEmpty, 60s)"| node
+ runner ---|"scheduled on"| node
+ sqs --> karpenter
+ runner -->|"EKS Pod Identity<br/>(scoped S3 only)"| s3
+```
+
+Terraform (`terraform/eks/`) creates everything under **AWS**; helmfile
+(`helm/infra-helmfile.yaml`, `helm/ci-helmfile.yaml.gotmpl`) deploys everything inside
+**EKS**; the `github-config` secret is created once by hand at bootstrap.
+
+- **Terraform** (this repo, `terraform/eks/`): VPC, EKS cluster, system node group,
+ Karpenter controller IAM (Pod Identity), node IAM role, spot-interruption SQS queue.
+- **Helmfile** (this repo, `helm/`): Karpenter, ARC controller, runner scale sets,
+ Karpenter `NodePool`/`EC2NodeClass` CRs, namespaces.
+- **Airflow repo**: `runs-on: airflow-runner-amd-xl` on the Tests (AMD) job graph
+ (experiment branch; later behind the existing `selective_checks` runner seam for
+ canary-only routing).
+
+## Key decisions (and what was rejected)
+
+### EKS standard mode + self-managed Karpenter — not EKS Auto Mode
+
+Auto Mode was seriously considered (it removes Karpenter/AMI/addon management) and
+rejected for this workload:
+
+- Auto Mode nodes are locked-down Bottlerocket: SELinux enforcing, immutable root FS,
+ no SSH/SSM. CI runners are privileged docker-in-docker pods that run KinD
+ (Kubernetes-in-Docker-in-Docker) and need node sysctls (`fs.inotify.*`,
+ `nf_conntrack`, `fs.nr_open`) — exactly the class of workload that fights a locked
+ host. Standard AL2023 nodes with our own user data are the proven path.
+- Auto Mode adds a per-instance management fee (~10–12% of the instance price) on a
+ budget-constrained, compute-dominated bill.
+- We already need Helm/CRD management for ARC; Karpenter via the same helmfile is
+ marginal extra complexity.
+
+### Karpenter v1.11 — not cluster-autoscaler + 6 static node groups
+
+The previous design (this branch pre-rebase) used six per-size/arch managed node
+groups, ASG label/taint tags for scale-from-zero (`asg_tags.tf`), and
+cluster-autoscaler. Karpenter replaces all of it:
+
+- Spot-price-aware instance selection across families (the "instance diversity"
+ discussion in PR #59 becomes a one-line requirement list).
+- Scale-from-zero without ASG tag gymnastics → `asg_tags.tf` deleted.
+- Consolidation and native spot-interruption handling (SQS) built in.
+- One `NodePool` instead of N node groups; an ARM pool later is ~10 lines.
+
+### ARC runner scale sets 0.14.x — still the supported approach in 2026
+
+`gha-runner-scale-set` (Autoscaling Runner Scale Sets mode) remains GitHub's
+maintained ARC mode (latest chart 0.14.2, 2026-05). Ephemeral runners, JIT
+registration, no cert-manager/webhook dependency (that was legacy ARC). Auth via a
+**GitHub App** (recommended over PAT): for repo-level runners it needs
+*Administration: R/W* + *Metadata: R*; for org-level, *Self-hosted runners: R/W*.
+
+### Utilities: what's in, what's out
+
+| Utility | Verdict | Why |
+|---|---|---|
+| Karpenter | **In** | Node autoscaling; replaces CA + static node groups; spot handling. Cost: negligible (runs on system nodes). |
+| ARC (scale sets) | **In** | The runner integration itself. |
+| metrics-server | **Out** | ARC scales on the job queue, not resource metrics; no HPA anywhere. Add only if we later want `kubectl top`. |
+| cert-manager | **Out** (removed from branch) | Only legacy-mode ARC needed it for admission webhooks. Scale-set mode doesn't. |
+| EBS CSI driver | **Out** (removed from branch) | No PersistentVolumes: runner work dirs + dind storage are node-local ephemeral (200 GiB gp3 root via `EC2NodeClass`). |
+| cluster-autoscaler | **Out** (replaced) | See Karpenter. |
+| External Secrets Operator | **Out** | Exactly one static secret (GitHub App key). ESO adds a controller + still needs a backing store + IAM. `kubectl create secret` at bootstrap is simpler and rotated rarely. Revisit if secrets multiply. |
+| AWS Load Balancer Controller | **Out** | Nothing is exposed; ARC listeners do outbound long-poll only. No Service/Ingress of type LB. |
+
+### Networking: public subnets, no NAT — kept deliberately
+
+Nodes launch in public subnets with public IPs (as in the original PR, which removed
+the NAT gateway). A NAT gateway costs ~$395/yr + data processing; public IPv4 on
+ephemeral nodes costs $0.005/h only while nodes exist (≈$40/yr at our duty cycle).
+Security posture is acceptable because: security groups allow no inbound; runners
+only make outbound calls (GitHub long-poll, image pulls); nodes use IMDSv2 with hop
+limit 1, so **pods (i.e. CI jobs) cannot reach node IAM credentials**; and the
+node role carries only EKS/ECR/CNI policies.
+
+### Compute: one runner class now, right-sizing later
+
+Measured mix of a full Tests (AMD) run (duration-weighted minutes by required vCPU
+class): `2:787, 4:209, 8:179, 16:96, 32:22, 48:6`. The cost-optimal design is 2–3
+scale-set size classes, but that requires per-job `runs-on` routing in the Airflow
+repo. To keep the Airflow diff minimal for the experiment, we start with a **single
+class sized so no job regresses** vs the previous baseline (32+ vCPU nodes):
+
+- Scale set `airflow-runner-amd-xl`: requests **28 vCPU / 56 Gi** → Karpenter picks
+ the cheapest current-gen ≥8xlarge (c6a/c7a/m6a/m7a/r6a/r7a, 32 vCPU) spot node;
+ one runner pod per node (requests-based, no anti-affinity hack).
+- Spot first with automatic on-demand fallback (`capacity-type: [spot, on-demand]`).
+ Consolidation is `WhenEmpty` only — running jobs are never evicted to repack.
+- maxRunners 100 (measured burst peak: 77 concurrent on the reduced run, ~163 on a
+ full canary; raise when canary-scale traffic is routed here).
+
+Cost at measured volume: ~21.7 node-hours per full run × ~$0.55–0.60/h (8xlarge
+spot) ≈ **$12–13/run** vs $104 before — before phase-2 right-sizing, which
+takes it to ~$4–5.
+
+**ARM later: yes, worth it.** Graviton spot is 10–25% cheaper per vCPU and the ARM
+canary exists. It's one extra `NodePool` (arch: arm64) + one scale set + an ARM
+runner image tag. Deliberately out of scope until AMD is proven.
+
+### dind configuration carries the lessons of the previous experiment
+
+The runner pods define an explicit dind sidecar (not the chart's default) so dockerd
+starts with flags that fix real failures seen in the previous experiment:
+
+- `--registry-mirror=https://mirror.gcr.io` — Docker Hub anonymous pull limits on
+ shared cloud egress IPs (also covers in-container `testcontainers` pulls).
+- `--default-address-pool base=172.16.0.0/14,size=24` — parallel breeze compose
+ networks exhaust Docker's default ~31 subnets.
+- `--default-ulimit nofile=65536:1048576` — venv-heavy tests exhaust fds; soft cap
+ kept moderate to avoid the close()-loop spawn trap.
+- cgroup-v2 delegation for KinD: `docker:dind`'s entrypoint handles this correctly
+ (the earlier failure existed precisely because that dockerd was *not* dind).
+- Node sysctls (`fs.inotify.max_user_watches/instances`, `nf_conntrack_tcp_be_liberal`,
+ `fs.nr_open`, `fs.file-max`) are set in `EC2NodeClass` user data.
+
+### Terraform layout: one root module, S3 backend
+
+Single root module at `terraform/eks/` (no premature module split — the AWS-side
+footprint is ~5 resources plus two registry modules). Existing S3 backend
+(`airflow-ci-tfstate` / DynamoDB `TerraformLocks`, us-east-2) is kept for the real
+deployment; it still holds the **stale 2024 state** of the deleted cluster (the
+`airflow-vpc` 10.1.0.0/16 VPC still exists in the account!) — reconcile before the
+first real apply (import-or-destroy; see README). Local iteration uses a
+`*_override.tf` local backend so `plan` runs without touching shared state. The
+provider no longer hardcodes `profile = "airflow"` — use `AWS_PROFILE`/env, which
+also works for a future GitHub Actions plan/apply workflow via OIDC role assumption.
+
+Versions: Terraform ≥ 1.9, AWS provider ~> 6.0, `terraform-aws-modules/eks` ~> 21.24
+(EKS 1.34, standard support to 2027-12), `terraform-aws-modules/vpc` ~> 6.6,
+Karpenter 1.11.3, ARC chart 0.14.2.
+
+### Security & trust boundary
+
+- Runners serve **trusted refs only**: scheduled/canary runs and (later) explicitly
+ label-gated committer PRs. Fork PRs stay on GitHub-hosted runners. For the org
+ deployment, the scale sets sit in a dedicated runner group restricted to
+ apache/airflow with "public repositories: selected workflows" disabled.
+- GitHub App over PAT: scoped, auditable, revocable; key lives only in the
+ `arc-runners` namespace secret.
+- Ephemeral runners: one job per pod, pod deleted after the job — no state bleed.
+- Jobs can't reach node IAM credentials (IMDS hop limit 1); the node role has no
+ access beyond EKS basics; the AWS account contains no Airflow production secrets.
+- Cluster API endpoint is public + access entries (admin = cluster creator +
+ explicitly added PMC operators); no aws-auth ConfigMap.
+
+## Scaling behavior
+
+1. Job queued on GitHub → ARC listener (long-poll) sees it, requests a runner pod.
+2. Pod is Pending (no capacity) → Karpenter launches the cheapest fitting spot
+ instance (~60–90 s to Ready including image pull).
+3. Job runs in the ephemeral runner (dind sidecar provides Docker).
+4. Pod completes and is deleted → node goes empty → Karpenter consolidates it away
+ after 60 s. Idle cluster = 2× t4g.medium + control plane ≈ **$1.6k/yr fixed**.
+5. Spot interruption → SQS → Karpenter drains/replaces; the interrupted job fails
+ and is retried by the workflow's normal retry/re-run mechanisms (acceptable for
+ canary; pin the pool to on-demand if measured interruption rates hurt).
+
+## Costs (measured workload, official prices)
+
+| | Previous experiment | EKS this design | EKS after right-sizing |
+|---|---|---|---|
+| Full Tests (AMD) run | $104 | ~$12–13 | ~$4–5 |
+| Scheduled-only, 2 AMD + 2 ARM per day, /yr | ~$135k+ | ~$20k | ~$9–11k |
+| Fixed (cluster + system nodes + misc) | — | ~$1.8–2.1k/yr | same |
+
+## What changes in the Airflow repo (experiment branch)
+
+Only the Tests (AMD) job graph: `runs-on` swapped from the previous label array to
+`airflow-runner-amd-xl`, and the self-hosted host guards extended to cover
+the ARC runner environment. Everything else (selective checks, breeze, tests) is
+untouched. The long-term mechanism is the existing `selective_checks` runner seam
+(static labels work with ARC, unlike the per-run label of the previous experiment), flipped for canary
+runs only.
diff --git a/helm/ci-helmfile.yaml b/helm/ci-helmfile.yaml.gotmpl
similarity index 76%
rename from helm/ci-helmfile.yaml
rename to helm/ci-helmfile.yaml.gotmpl
index e1da5a0..c707c9f 100644
--- a/helm/ci-helmfile.yaml
+++ b/helm/ci-helmfile.yaml.gotmpl
@@ -20,19 +20,26 @@ environments:
values:
- ./values/gha-runner-scale-sets/runners.yaml
+# helmfile v1 requires environments and releases in separate YAML documents.
+---
releases:
- name: actions-runner-controller
chart: oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
- namespace: ci
- version: 0.9.3
+ namespace: arc-systems
+ version: 0.14.2
values:
- ./values/gha-runner-scale-set-controller.yaml
{{- range $key, $value := .Values.runnerScaleSets }}
- name: {{ $key }}
chart: oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set
- namespace: ci
- version: 0.9.3
+ namespace: arc-runners
+ version: 0.14.2
+ needs:
+ - arc-systems/actions-runner-controller
+ # AutoscalingRunnerSet CRD ships with the controller chart above — skip
+ # upfront validation so a fresh cluster can install in one pass.
+ disableValidationOnInstall: true
values:
- ./values/gha-runner-scale-sets/common.yaml.gotmpl
{{- end }}
diff --git a/helm/cluster-config/templates/namespaces/infra.yaml b/helm/cluster-config/templates/namespaces/arc.yaml
similarity index 85%
rename from helm/cluster-config/templates/namespaces/infra.yaml
rename to helm/cluster-config/templates/namespaces/arc.yaml
index f6bd908..50450f3 100644
--- a/helm/cluster-config/templates/namespaces/infra.yaml
+++ b/helm/cluster-config/templates/namespaces/arc.yaml
@@ -18,9 +18,14 @@
*/}}
################################################################################
-# infra namespace where we create the infrastructure resources
+# arc-systems: ARC controller + listeners; arc-runners: runner pods + secrets
################################################################################
apiVersion: v1
kind: Namespace
metadata:
- name: infra
+ name: arc-systems
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+ name: arc-runners
diff --git a/helm/cluster-config/templates/namespaces/ci.yaml b/helm/cluster-config/templates/namespaces/ci.yaml
deleted file mode 100644
index 123ddc1..0000000
--- a/helm/cluster-config/templates/namespaces/ci.yaml
+++ /dev/null
@@ -1,26 +0,0 @@
-{{/*
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-*/}}
-
-################################################################################
-# ci namespace where we create the CI resources
-################################################################################
-apiVersion: v1
-kind: Namespace
-metadata:
- name: ci
diff --git a/helm/infra-helmfile.yaml b/helm/infra-helmfile.yaml
index d8c49e5..100dd84 100644
--- a/helm/infra-helmfile.yaml
+++ b/helm/infra-helmfile.yaml
@@ -15,38 +15,26 @@
# specific language governing permissions and limitations
# under the License.
---
-repositories:
- - name: jetstack
- url: https://charts.jetstack.io
- - name: autoscaler
- url: https://kubernetes.github.io/autoscaler
- - name: aws-ebs-csi-driver
- url: https://kubernetes-sigs.github.io/aws-ebs-csi-driver
-
releases:
- name: cluster-config
chart: ./cluster-config
namespace: kube-system
- # https://github.com/kubernetes-sigs/aws-ebs-csi-driver/tree/master/charts/aws-ebs-csi-driver
- - name: aws-ebs-csi-driver
- chart: aws-ebs-csi-driver/aws-ebs-csi-driver
+ # https://karpenter.sh/docs/getting-started/
+ # IAM/SQS for Karpenter comes from terraform/eks/karpenter.tf (Pod Identity).
+ - name: karpenter
+ chart: oci://public.ecr.aws/karpenter/karpenter
namespace: kube-system
+ version: 1.11.3
values:
- - ./values/ebs-csi-dirver.yaml
-
- # https://cert-manager.io/docs/installation/helm/
- - name: cert-manager
- chart: jetstack/cert-manager
- namespace: infra
- version: v1.15.3
- values:
- - ./values/cert-manager.yaml
+ - ./values/karpenter.yaml
- # https://github.com/kubernetes/autoscaler/tree/master/charts/cluster-autoscaler
- - name: autoscaler
- chart: autoscaler/cluster-autoscaler
- namespace: infra
- version: 9.37.0
- values:
- - ./values/cluster-autoscaler.yaml
+ # NodePool/EC2NodeClass CRs — depends on the Karpenter CRDs.
+ - name: karpenter-resources
+ chart: ./karpenter-resources
+ namespace: kube-system
+ needs:
+ - kube-system/karpenter
+ # On a fresh cluster the CRDs don't exist until the karpenter release above
+ # installs, so the upfront diff/validation of this release must be skipped.
+ disableValidationOnInstall: true
diff --git a/helm/values/ebs-csi-dirver.yaml b/helm/karpenter-resources/Chart.yaml
similarity index 83%
rename from helm/values/ebs-csi-dirver.yaml
rename to helm/karpenter-resources/Chart.yaml
index 7f8f0d0..e33e60d 100644
--- a/helm/values/ebs-csi-dirver.yaml
+++ b/helm/karpenter-resources/Chart.yaml
@@ -15,7 +15,8 @@
# specific language governing permissions and limitations
# under the License.
---
-controller:
- serviceAccount:
- annotations:
- eks.amazonaws.com/role-arn: arn:aws:iam::827901512104:role/ebs-csi-driver
+apiVersion: v2
+name: karpenter-resources
+description: Karpenter NodePools and EC2NodeClasses for the Airflow CI runners
+type: application
+version: 0.1.0
diff --git a/helm/karpenter-resources/templates/ec2nodeclass.yaml b/helm/karpenter-resources/templates/ec2nodeclass.yaml
new file mode 100644
index 0000000..b5058dd
--- /dev/null
+++ b/helm/karpenter-resources/templates/ec2nodeclass.yaml
@@ -0,0 +1,71 @@
+{{/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+*/}}
+apiVersion: karpenter.k8s.aws/v1
+kind: EC2NodeClass
+metadata:
+ name: ci
+spec:
+ amiSelectorTerms:
+ - alias: al2023@latest
+ role: {{ .Values.nodeIamRoleName }}
+ subnetSelectorTerms:
+ - tags:
+ karpenter.sh/discovery: {{ .Values.clusterName }}
+ securityGroupSelectorTerms:
+ - tags:
+ karpenter.sh/discovery: {{ .Values.clusterName }}
+ associatePublicIPAddress: true
+ metadataOptions:
+ httpTokens: required
+ # Hop limit 1 keeps node IAM credentials unreachable from the runner
+ # containers (CI jobs must not get AWS access via IMDS).
+ httpPutResponseHopLimit: 1
+ blockDeviceMappings:
+ - deviceName: /dev/xvda
+ ebs:
+ volumeSize: {{ .Values.nodeVolumeSize }}
+ volumeType: gp3
+ iops: 4000
+ throughput: 250
+ encrypted: true
+ deleteOnTermination: true
+ userData: |
+ MIME-Version: 1.0
+ Content-Type: multipart/mixed; boundary="BOUNDARY"
+
+ --BOUNDARY
+ Content-Type: text/x-shellscript; charset="us-ascii"
+
+ #!/bin/bash
+ # Kernel limits for KinD clusters and fd-heavy test suites; GitHub-hosted
+ # runners ship these, stock AL2023 does not.
+ modprobe nf_conntrack || true
+ sysctl -w fs.inotify.max_user_watches=524288
+ sysctl -w fs.inotify.max_user_instances=512
+ sysctl -w net.netfilter.nf_conntrack_tcp_be_liberal=1 || true
+ sysctl -w fs.nr_open=2097152
+ sysctl -w fs.file-max=2097152
+ # Disable core dumps: AL2023's default systemd-coredump handler is
+ # kernel-global, so a SEGV inside any container dumps the whole process
+ # through the node's journald first — under test-suite disk load that
+ # takes >60s and broke task-sdk's test_exit_by_signal[segv] (GitHub's
+ # Ubuntu runners short-circuit via apport instead). CI nodes are
+ # ephemeral; nobody reads their dumps.
+ sysctl -w "kernel.core_pattern=|/bin/false"
+ --BOUNDARY--
diff --git a/helm/karpenter-resources/templates/nodepool-amd-shared.yaml b/helm/karpenter-resources/templates/nodepool-amd-shared.yaml
new file mode 100644
index 0000000..b2f2eab
--- /dev/null
+++ b/helm/karpenter-resources/templates/nodepool-amd-shared.yaml
@@ -0,0 +1,65 @@
+{{/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+*/}}
+{{/*
+ On-demand pool for the bin-packed small/medium runner classes. Shared nodes
+ concentrate many jobs, so a single spot reclaim kills them all at once and a
+ rerun re-packs the same jobs into the same vulnerable shape (measured
+ 2026-07-27/28: reclaim storms of 5-57 events wiped whole retry batches
+ repeatedly; the on-demand premium is ~$1.5-2 per run). The xl class stays on
+ the spot-first `ci-amd` pool: one job per node, so reclaims stay cheap,
+ independent retries.
+*/}}
+apiVersion: karpenter.sh/v1
+kind: NodePool
+metadata:
+ name: ci-amd-shared
+spec:
+ template:
+ metadata:
+ labels:
+ airflow-ci/pool: amd-shared
+ spec:
+ nodeClassRef:
+ group: karpenter.k8s.aws
+ kind: EC2NodeClass
+ name: ci
+ taints:
+ - key: airflow-ci/runner
+ effect: NoSchedule
+ requirements:
+ - key: kubernetes.io/arch
+ operator: In
+ values: ["amd64"]
+ - key: kubernetes.io/os
+ operator: In
+ values: ["linux"]
+ - key: karpenter.sh/capacity-type
+ operator: In
+ values: ["on-demand"]
+ - key: karpenter.k8s.aws/instance-family
+ operator: In
+ values: ["c6a", "c7a", "m6a", "m7a", "r6a", "r7a", "c6i", "c7i", "m6i", "m7i"]
+ - key: karpenter.k8s.aws/instance-cpu
+ operator: In
+ values: ["8", "16", "32", "48"]
+ disruption:
+ consolidationPolicy: WhenEmpty
+ consolidateAfter: 60s
+ limits:
+ cpu: {{ .Values.amdSharedPoolCpuLimit | quote }}
diff --git a/helm/karpenter-resources/templates/nodepool-amd.yaml b/helm/karpenter-resources/templates/nodepool-amd.yaml
new file mode 100644
index 0000000..b053dba
--- /dev/null
+++ b/helm/karpenter-resources/templates/nodepool-amd.yaml
@@ -0,0 +1,63 @@
+{{/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+*/}}
+apiVersion: karpenter.sh/v1
+kind: NodePool
+metadata:
+ name: ci-amd
+spec:
+ template:
+ metadata:
+ labels:
+ airflow-ci/pool: amd
+ spec:
+ nodeClassRef:
+ group: karpenter.k8s.aws
+ kind: EC2NodeClass
+ name: ci
+ # Only runner pods (which tolerate this) land on these nodes, so they
+ # never pin cluster infra and can be reclaimed the moment a job ends.
+ taints:
+ - key: airflow-ci/runner
+ effect: NoSchedule
+ requirements:
+ - key: kubernetes.io/arch
+ operator: In
+ values: ["amd64"]
+ - key: kubernetes.io/os
+ operator: In
+ values: ["linux"]
+ # Spot first; Karpenter falls back to on-demand when spot is unavailable.
+ - key: karpenter.sh/capacity-type
+ operator: In
+ values: ["spot", "on-demand"]
+ - key: karpenter.k8s.aws/instance-family
+ operator: In
+ values: ["c6a", "c7a", "m6a", "m7a", "r6a", "r7a", "c6i", "c7i", "m6i", "m7i"]
+ # Small enough sizes for the small/medium runner classes to get
+ # right-sized (or bin-packed) nodes; xl pods still land on >= 32 vCPU
+ # because their 28-vCPU request cannot fit anything smaller.
+ - key: karpenter.k8s.aws/instance-cpu
+ operator: In
+ values: ["8", "16", "32", "48"]
+ disruption:
+ # Never repack running CI jobs; only reap empty nodes.
+ consolidationPolicy: WhenEmpty
+ consolidateAfter: 60s
+ limits:
+ cpu: {{ .Values.amdPoolCpuLimit | quote }}
diff --git a/helm/karpenter-resources/templates/nodepool-arm-shared.yaml b/helm/karpenter-resources/templates/nodepool-arm-shared.yaml
new file mode 100644
index 0000000..b1c402b
--- /dev/null
+++ b/helm/karpenter-resources/templates/nodepool-arm-shared.yaml
@@ -0,0 +1,65 @@
+{{/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+*/}}
+{{/*
+ On-demand pool for the bin-packed small/medium runner classes. Shared nodes
+ concentrate many jobs, so a single spot reclaim kills them all at once and a
+ rerun re-packs the same jobs into the same vulnerable shape (measured
+ 2026-07-27/28: reclaim storms of 5-57 events wiped whole retry batches
+ repeatedly; the on-demand premium is ~$1.5-2 per run). The xl class stays on
+ the spot-first `ci-arm` pool: one job per node, so reclaims stay cheap,
+ independent retries.
+*/}}
+apiVersion: karpenter.sh/v1
+kind: NodePool
+metadata:
+ name: ci-arm-shared
+spec:
+ template:
+ metadata:
+ labels:
+ airflow-ci/pool: arm-shared
+ spec:
+ nodeClassRef:
+ group: karpenter.k8s.aws
+ kind: EC2NodeClass
+ name: ci
+ taints:
+ - key: airflow-ci/runner
+ effect: NoSchedule
+ requirements:
+ - key: kubernetes.io/arch
+ operator: In
+ values: ["arm64"]
+ - key: kubernetes.io/os
+ operator: In
+ values: ["linux"]
+ - key: karpenter.sh/capacity-type
+ operator: In
+ values: ["on-demand"]
+ - key: karpenter.k8s.aws/instance-family
+ operator: In
+ values: ["c6g", "c7g", "c8g", "m6g", "m7g", "m8g", "r6g", "r7g", "r8g"]
+ - key: karpenter.k8s.aws/instance-cpu
+ operator: In
+ values: ["8", "16", "32", "48"]
+ disruption:
+ consolidationPolicy: WhenEmpty
+ consolidateAfter: 60s
+ limits:
+ cpu: {{ .Values.armSharedPoolCpuLimit | quote }}
diff --git a/helm/karpenter-resources/templates/nodepool-arm.yaml b/helm/karpenter-resources/templates/nodepool-arm.yaml
new file mode 100644
index 0000000..2d6e763
--- /dev/null
+++ b/helm/karpenter-resources/templates/nodepool-arm.yaml
@@ -0,0 +1,63 @@
+{{/*
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+*/}}
+apiVersion: karpenter.sh/v1
+kind: NodePool
+metadata:
+ name: ci-arm
+spec:
+ template:
+ metadata:
+ labels:
+ airflow-ci/pool: arm
+ spec:
+ nodeClassRef:
+ group: karpenter.k8s.aws
+ kind: EC2NodeClass
+ name: ci
+ # Only runner pods (which tolerate this) land on these nodes, so they
+ # never pin cluster infra and can be reclaimed the moment a job ends.
+ taints:
+ - key: airflow-ci/runner
+ effect: NoSchedule
+ requirements:
+ - key: kubernetes.io/arch
+ operator: In
+ values: ["arm64"]
+ - key: kubernetes.io/os
+ operator: In
+ values: ["linux"]
+ # Spot first; Karpenter falls back to on-demand when spot is unavailable.
+ - key: karpenter.sh/capacity-type
+ operator: In
+ values: ["spot", "on-demand"]
+ - key: karpenter.k8s.aws/instance-family
+ operator: In
+ values: ["c6g", "c7g", "c8g", "m6g", "m7g", "m8g", "r6g", "r7g", "r8g"]
+ # Small enough sizes for the small/medium runner classes to get
+ # right-sized (or bin-packed) nodes; xl pods still land on >= 32 vCPU
+ # because their 28-vCPU request cannot fit anything smaller.
+ - key: karpenter.k8s.aws/instance-cpu
+ operator: In
+ values: ["8", "16", "32", "48"]
+ disruption:
+ # Never repack running CI jobs; only reap empty nodes.
+ consolidationPolicy: WhenEmpty
+ consolidateAfter: 60s
+ limits:
+ cpu: {{ .Values.armPoolCpuLimit | quote }}
diff --git a/helm/values/cluster-autoscaler.yaml b/helm/karpenter-resources/values.yaml
similarity index 51%
copy from helm/values/cluster-autoscaler.yaml
copy to helm/karpenter-resources/values.yaml
index 2b4bf57..295db57 100644
--- a/helm/values/cluster-autoscaler.yaml
+++ b/helm/karpenter-resources/values.yaml
@@ -15,21 +15,21 @@
# specific language governing permissions and limitations
# under the License.
---
-cloudProvider: aws
-awsRegion: us-east-2
+clusterName: airflow-ci
-autoDiscovery:
- clusterName: "airflow"
+# Must match node_iam_role_name in terraform/eks/karpenter.tf.
+nodeIamRoleName: airflow-ci-runner-node
-rbac:
- create: true
- serviceAccount:
- create: true
- name: "cluster-autoscaler"
- annotations:
- eks.amazonaws.com/role-arn: arn:aws:iam::827901512104:role/eks-autoscaler-role
+# Root volume for runner nodes: CI image + KinD images + parallel compose
+# stacks need far more than the 20Gi AL2023 default (the previous runners had 256GB).
+nodeVolumeSize: 200Gi
-extraArgs:
- scan-interval: 10s
- scale-down-delay-after-add: 5m
- scale-down-unneeded-time: 5m
+# Upper bound on total simultaneously provisioned runner capacity
+# (~100 concurrent 32-vCPU runner nodes).
+amdPoolCpuLimit: "3200"
+# On-demand shared pool (small/medium classes): sized for the measured
+# full-matrix peak (~60 nodes averaging well under 48 vCPU each).
+amdSharedPoolCpuLimit: "2000"
+# ARM (Graviton) pools mirror the AMD split; experiment-sized like amd was.
+armPoolCpuLimit: "3200"
+armSharedPoolCpuLimit: "2000"
diff --git a/helm/values/cert-manager.yaml b/helm/values/cert-manager.yaml
deleted file mode 100644
index be548e5..0000000
--- a/helm/values/cert-manager.yaml
+++ /dev/null
@@ -1,20 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
----
-installCRDs: true
-nodeSelector:
- node-type: default
diff --git a/helm/values/gha-runner-scale-set-controller.yaml b/helm/values/gha-runner-scale-set-controller.yaml
index 63ab994..1ed8177 100644
--- a/helm/values/gha-runner-scale-set-controller.yaml
+++ b/helm/values/gha-runner-scale-set-controller.yaml
@@ -18,8 +18,7 @@
replicaCount: 1
labels:
- type: ci
app: gha-runner-scale-set-controller
-nodeSelector:
- node-type: default
+# Watch all namespaces (scale sets live in arc-runners); no nodeSelector needed:
+# runner nodes are tainted, so the controller lands on the system node group.
diff --git a/helm/values/gha-runner-scale-sets/common.yaml.gotmpl b/helm/values/gha-runner-scale-sets/common.yaml.gotmpl
index 9ed5238..7013587 100644
--- a/helm/values/gha-runner-scale-sets/common.yaml.gotmpl
+++ b/helm/values/gha-runner-scale-sets/common.yaml.gotmpl
@@ -17,45 +17,134 @@
---
{{ $config := index .Values.runnerScaleSets .Release.Name }}
-githubConfigUrl: https://github.com/apache
+githubConfigUrl: {{ .Values.githubConfigUrl }}
+# Pre-created secret in the arc-runners namespace holding the GitHub App
+# credentials (github_app_id / github_app_installation_id / github_app_private_key).
githubConfigSecret: github-config
-runnerGroup: airflow-ci
-
minRunners: {{ $config.minRunners }}
maxRunners: {{ $config.maxRunners }}
-containerMode:
- type: "dind"
-
+# The dind sidecar is spelled out (instead of `containerMode: dind`) so dockerd
+# gets flags that fix real failures measured on the previous self-hosted runners:
+# --registry-mirror Docker Hub anonymous pull limits on shared cloud
+# egress IPs (also covers in-container testcontainers
+# pulls that cannot use host credentials).
+# --default-address-pool parallel breeze compose stacks exhaust Docker's
+# default ~31 network pools; /14 carved into /24s
+# gives 1024 networks. Must not overlap the VPC CIDR
+# (10.1.0.0/16).
+# --default-ulimit nofile fd-heavy venv tests need >1024; the soft cap stays
+# at 65536 because processes that close(0..soft) on
+# spawn make a high soft limit a performance trap.
template:
+ metadata:
+ annotations:
+ # Karpenter must never drain a node out from under a running CI job —
+ # not for drift (e.g. EC2NodeClass changes) and not for consolidation.
+ # Involuntary events (spot reclaim) are unaffected.
+ karpenter.sh/do-not-disrupt: "true"
spec:
+ # CI jobs must not see this cluster's credentials: with the token mounted,
+ # kubernetes clients inside tests prefer in-cluster config over the KinD
+ # kubeconfig and hit the EKS API as the runner SA (403s broke the K8S
+ # system-test teardowns). EKS Pod Identity (S3 stash) uses its own
+ # separately-projected token and is unaffected.
+ automountServiceAccountToken: false
+ initContainers:
+ - name: init-dind-externals
+ image: {{ .Values.image }}
+ command: ["cp", "-r", "/home/runner/externals/.", "/home/runner/tmpDir/"]
+ volumeMounts:
+ - name: dind-externals
+ mountPath: /home/runner/tmpDir
+ - name: dind
+ image: docker:dind
+ args:
+ - dockerd
+ - --host=unix:///var/run/docker.sock
+ - --group=$(DOCKER_GROUP_GID)
+ - --registry-mirror=https://mirror.gcr.io
+ - --default-address-pool=base=172.16.0.0/14,size=24
+ - --default-ulimit=nofile=65536:1048576
+ # Classic image store (overlay2), matching GitHub-hosted runners.
+ # The containerd store (dind default) breaks `kind load docker-image`:
+ # docker save emits OCI manifests whose blobs ctr import --digests
+ # can't resolve ("content digest ... not found").
+ - --feature=containerd-snapshotter=false
+ env:
+ - name: DOCKER_GROUP_GID
+ value: "123"
+ securityContext:
+ privileged: true
+ restartPolicy: Always
+ startupProbe:
+ exec:
+ command: ["docker", "info"]
+ periodSeconds: 5
+ failureThreshold: 24
+ volumeMounts:
+ - name: work
+ mountPath: /home/runner/_work
+ - name: dind-sock
+ mountPath: /var/run
+ - name: dind-externals
+ mountPath: /home/runner/externals
+ # /tmp is shared with the runner (see runner container) so that
+ # `docker run -v /tmp/...` host paths resolve — the daemon lives
+ # here, not in the runner container.
+ - name: shared-tmp
+ mountPath: /tmp
containers:
- name: runner
image: {{ .Values.image }}
- command: [ "/home/runner/run.sh" ]
+ command: ["/home/runner/run.sh"]
+ env:
+ - name: DOCKER_HOST
+ value: unix:///var/run/docker.sock
+ - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
+ value: "120"
+ # Lets airflow-repo CI scripts detect the self-hosted K8s runner
+ # (guards for host-specific steps like the /mnt docker relocation).
+ - name: AIRFLOW_SELF_HOSTED_RUNNER
+ value: "eks"
+ # Class-sized xdist worker cap; run_unit_tests.sh only defaults
+ # PARALLELISM when unset, so this wins over its self-hosted fallback.
+ - name: PARALLELISM
+ value: {{ $config.parallelism | quote }}
+ resources:
+ requests:
+ cpu: {{ $config.cpu | quote }}
+ memory: {{ $config.memory }}
+ volumeMounts:
+ - name: work
+ mountPath: /home/runner/_work
+ - name: dind-sock
+ mountPath: /var/run
+ # Shared with the dind sidecar: tests bind-mount pytest tmpdirs
+ # (docker run -v /tmp/...) and the daemon resolves those paths in
+ # ITS filesystem. On GitHub-hosted runners daemon and job share one
+ # host; this emptyDir recreates that contract for /tmp.
+ - name: shared-tmp
+ mountPath: /tmp
+ volumes:
+ - name: work
+ emptyDir: {}
+ - name: dind-sock
+ emptyDir: {}
+ - name: dind-externals
+ emptyDir: {}
+ - name: shared-tmp
+ emptyDir: {}
nodeSelector:
- node-type: gha-runners
- size: {{ $config.size }}
- arch: {{ $config.arch }}
+ airflow-ci/pool: {{ $config.pool }}
tolerations:
- - key: node-type
- operator: Equal
- value: gha-runners
+ - key: airflow-ci/runner
+ operator: Exists
effect: NoSchedule
- affinity:
- podAntiAffinity:
- requiredDuringSchedulingIgnoredDuringExecution:
- - labelSelector:
- matchExpressions:
- - key: actions.github.com/scale-set-name
- operator: In
- values:
- - {{ .Release.Name }}
- topologyKey: kubernetes.io/hostname
controllerServiceAccount:
name: actions-runner-controller-gha-rs-controller
- namespace: {{ .Release.Namespace }}
+ namespace: arc-systems
diff --git a/helm/values/gha-runner-scale-sets/runners.yaml b/helm/values/gha-runner-scale-sets/runners.yaml
index 3c475fe..4d86536 100644
--- a/helm/values/gha-runner-scale-sets/runners.yaml
+++ b/helm/values/gha-runner-scale-sets/runners.yaml
@@ -15,35 +15,70 @@
# specific language governing permissions and limitations
# under the License.
---
-image: ghcr.io/apache/airflow-ci-infra/actions-runner:20240902-rc1
+# Always a date-pinned tag, never :latest — an unreviewed image change must not
+# be able to break every canary at once. Rebuild monthly or when ARC's minimum
+# runner version forces it.
+# Fork-experiment override (public package on the fork owner's ghcr); switch to
+# ghcr.io/apache/airflow-ci-infra/actions-runner:<tag> for the production deploy.
+image: ghcr.io/shahar1/airflow-ci-actions-runner:20260731.2
+
+# Repo URL for the fork experiment; switch to https://github.com/apache
+# (org-level, with a restricted runner group) for the production deployment.
+githubConfigUrl: https://github.com/shahar1/airflow
+
+# Three size classes, assigned per job from the measured per-job peaks of the
+# 2026-07-26 go/no-go runs (airflow repo files/eks-class-mapping.md): each job's
+# runs-on picks the smallest class fitting peak x 1.3 headroom on both CPU and
+# memory (Non-DB providers is memory-bound at 35 GiB, hence xl). The xl request
+# fills a whole 32-vCPU node (one job = one node); small/medium pods bin-pack
+# when Karpenter batches several pending jobs onto one larger spot node.
+# parallelism feeds the PARALLELISM env (xdist worker cap in run_unit_tests.sh)
+# so test parallelism matches the class instead of oversubscribing small nodes.
runnerScaleSets:
- arc-small-amd:
- minRunners: 0
- maxRunners: 30
- size: small
- arch: x64
- arc-medium-amd:
- minRunners: 0
- maxRunners: 30
- size: medium
- arch: x64
- arc-large-amd:
- minRunners: 0
- maxRunners: 30
- size: large
- arch: x64
- arc-small-arm:
- minRunners: 0
- maxRunners: 30
- size: small
- arch: arm64
- arc-medium-arm:
- minRunners: 0
- maxRunners: 30
- size: medium
- arch: arm64
- arc-large-arm:
- minRunners: 0
- maxRunners: 30
- size: large
- arch: arm64
+ # small/medium bin-pack onto shared nodes -> on-demand pool (a spot reclaim
+ # of a shared node kills every job on it and reruns re-pack the same batch;
+ # see nodepool-amd-shared.yaml).
+ airflow-runner-amd-small:
+ minRunners: 0
+ maxRunners: 60
+ pool: amd-shared
+ cpu: "4"
+ memory: 16Gi
+ parallelism: "4"
+ airflow-runner-amd-medium:
+ minRunners: 0
+ maxRunners: 80
+ pool: amd-shared
+ cpu: "8"
+ memory: 32Gi
+ parallelism: "8"
+ airflow-runner-amd-xl:
+ minRunners: 0
+ maxRunners: 100
+ pool: amd
+ cpu: "28"
+ memory: 56Gi
+ parallelism: "16"
+ # ARM (Graviton) classes mirror the AMD layout; the image tag is a
+ # multi-arch manifest so the same reference serves both architectures.
+ airflow-runner-arm-small:
+ minRunners: 0
+ maxRunners: 60
+ pool: arm-shared
+ cpu: "4"
+ memory: 16Gi
+ parallelism: "4"
+ airflow-runner-arm-medium:
+ minRunners: 0
+ maxRunners: 80
+ pool: arm-shared
+ cpu: "8"
+ memory: 32Gi
+ parallelism: "8"
+ airflow-runner-arm-xl:
+ minRunners: 0
+ maxRunners: 100
+ pool: arm
+ cpu: "28"
+ memory: 56Gi
+ parallelism: "16"
diff --git a/helm/values/cluster-autoscaler.yaml b/helm/values/karpenter.yaml
similarity index 68%
rename from helm/values/cluster-autoscaler.yaml
rename to helm/values/karpenter.yaml
index 2b4bf57..4c5780f 100644
--- a/helm/values/cluster-autoscaler.yaml
+++ b/helm/values/karpenter.yaml
@@ -15,21 +15,17 @@
# specific language governing permissions and limitations
# under the License.
---
-cloudProvider: aws
-awsRegion: us-east-2
+settings:
+ clusterName: airflow-ci
+ # Must match queue_name in terraform/eks/karpenter.tf.
+ interruptionQueue: airflow-ci-karpenter
-autoDiscovery:
- clusterName: "airflow"
+# IAM comes from the EKS Pod Identity association created by Terraform —
+# no serviceAccount role annotation needed.
-rbac:
- create: true
- serviceAccount:
- create: true
- name: "cluster-autoscaler"
- annotations:
- eks.amazonaws.com/role-arn: arn:aws:iam::827901512104:role/eks-autoscaler-role
-
-extraArgs:
- scan-interval: 10s
- scale-down-delay-after-add: 5m
- scale-down-unneeded-time: 5m
+# Fits the 2x t4g.medium system nodes.
+controller:
+ resources:
+ requests:
+ cpu: 500m
+ memory: 768Mi
diff --git a/runner/Dockerfile b/runner/Dockerfile
index 683028a..6fc9126 100644
--- a/runner/Dockerfile
+++ b/runner/Dockerfile
@@ -19,17 +19,97 @@ FROM ghcr.io/actions/actions-runner:latest
USER root
+# Host-level tools the airflow CI workflows expect on a runner — the target
+# contract is GitHub's hosted ubuntu-24.04 image, restricted to what the CI
+# graph actually invokes (pip, uv-via-pip, aws, parallel, docker CLI). Jobs
+# also run `sudo apt-get install ...` themselves (as on GitHub-hosted
+# runners), so the runner user gets passwordless sudo instead of us baking in
+# every package.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
- ca-certificates curl nodejs npm wget unzip vim git jq build-essential netcat subversion \
+ sudo ca-certificates curl wget zip unzip zstd git jq parallel build-essential netcat-openbsd \
libssl-dev libffi-dev python3 python3-venv python3-dev python3-pip python-is-python3 \
- && npm install -g yarn \
&& install -m 0755 -d /etc/apt/keyrings \
&& curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc \
&& chmod a+r /etc/apt/keyrings/docker.asc \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null \
&& apt-get update \
- && apt-get install -y docker-compose-plugin
+ && apt-get install -y --no-install-recommends docker-ce-cli docker-compose-plugin docker-buildx-plugin \
+ && rm -rf /var/lib/apt/lists/* \
+ && echo "runner ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/runner \
+ && chmod 0440 /etc/sudoers.d/runner
-USER runner
\ No newline at end of file
+# The actions-runner base ships a malformed tz config (/etc/timezone = "/UTC",
+# localtime -> zoneinfo//UTC) which breaks local-timezone detection with
+# 'ZoneInfo keys may not be absolute paths' (seen in shared timezones tests).
+# Normalize to what GitHub-hosted runners have.
+RUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime && echo "Etc/UTC" > /etc/timezone
+
+# Same mechanism GitHub's runner image uses to make system `pip install` work
+# on Ubuntu 24.04 despite PEP 668 (actions/runner-images install-python.sh):
+# a global pip.conf, not deleting the EXTERNALLY-MANAGED marker. Non-root pip
+# installs fall through to ~/.local, hence the PATH addition.
+RUN printf '[global]\nbreak-system-packages = true\n' > /etc/pip.conf
+
+# JDKs for the java-sdk prek hooks (sync-java-sdk-supervisor-schema, ktlint,
+# spotless run gradle on the host; toolchains reference Java 11 and 17).
+# GitHub-hosted runners ship the Temurin set — mirror the versions we use.
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends openjdk-11-jdk-headless openjdk-17-jdk-headless \
+ && rm -rf /var/lib/apt/lists/* \
+ && ln -s "/usr/lib/jvm/java-17-openjdk-$(dpkg --print-architecture)" /usr/lib/jvm/java-17
+ENV JAVA_HOME=/usr/lib/jvm/java-17
+
+# gh CLI (job-summary and finalize steps use `gh api`/`gh run view`; present
+# on GitHub-hosted runners). Official apt repo per cli.github.com docs.
+RUN curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg -o /etc/apt/keyrings/githubcli-archive-keyring.gpg \
+ && chmod a+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \
+ && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" > /etc/apt/sources.list.d/github-cli.list \
+ && apt-get update \
+ && apt-get install -y --no-install-recommends gh \
+ && rm -rf /var/lib/apt/lists/*
+
+# kubectl (hosted runners ship it; the k8s workflows' cluster-state dump steps
+# call it directly on the host). Pinned to the cluster's minor via pkgs.k8s.io.
+RUN curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key -o /tmp/k8s.key \
+ && gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg /tmp/k8s.key \
+ && chmod a+r /etc/apt/keyrings/kubernetes-apt-keyring.gpg \
+ && echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /" > /etc/apt/sources.list.d/kubernetes.list \
+ && apt-get update \
+ && apt-get install -y --no-install-recommends kubectl \
+ && rm -rf /var/lib/apt/lists/* /tmp/k8s.key
+
+# Real python3.10 on PATH (uv-managed build; hosted runners ship toolcache
+# pythons, so this matches the contract). Provider-compat jobs check out old
+# airflow branches whose prek configs pin python3.10 hooks; without a local
+# interpreter prek downloads its own managed CPython, and that download is
+# broken on arm64 (sysconfigdata missing build_time_vars, first ARM run
+# 2026-07-31). A PATH python short-circuits the download on both arches.
+RUN pip install uv \
+ && UV_PYTHON_INSTALL_DIR=/opt/uv-python uv python install 3.10 \
+ && ln -s /opt/uv-python/cpython-3.10.*/bin/python3.10 /usr/local/bin/python3.10 \
+ && /usr/local/bin/python3.10 -c "import sysconfig; sysconfig.get_path('stdlib')" \
+ && pip uninstall -y uv
+
+# AWS CLI v2 (stash actions use `aws s3`); official installer, arch-aware.
+RUN ARCH=$(uname -m) \
+ && curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-${ARCH}.zip" -o /tmp/awscliv2.zip \
+ && unzip -q /tmp/awscliv2.zip -d /tmp \
+ && /tmp/aws/install \
+ && rm -rf /tmp/aws /tmp/awscliv2.zip
+
+ENV PATH="/home/runner/.local/bin:${PATH}"
+
+# Hosted runners export USER; workflows rely on it (e.g. `sudo chown -R
+# "${USER}"` — with USER unset that chown is a silent no-op, exit 0, leaving
+# root-owned dirs that later non-sudo steps can't write).
+ENV USER=runner
+
+# breeze saves/loads CI image tars via `--image-file-dir /mnt` — on GitHub-hosted
+# runners /mnt is the big scratch disk made writable by make_mnt_writeable.sh,
+# which self-hosted runs deliberately skip. Backed here by the node's 200 GB
+# root volume; sticky like /tmp.
+RUN mkdir -p /mnt && chmod 1777 /mnt
+
+USER runner
diff --git a/terraform/README.rst b/terraform/README.rst
index ad8e00b..e9832c3 100644
--- a/terraform/README.rst
+++ b/terraform/README.rst
@@ -28,23 +28,37 @@ Requirements
In order to deploy the infrastructure, you need to have the following tools installed:
-- `tfenv <https://github.com/tfutils/tfenv>`_ to manage Terraform versions
+- Terraform >= 1.9 (e.g. via `tfenv <https://github.com/tfutils/tfenv>`_: ``tfenv install 1.9.5``)
- Configure your AWS and EKS credentials :ref:`doc <aws_configuration>`
+State
+~~~~~
-Configure Terraform
-~~~~~~~~~~~~~~~~~~~
+The remote state lives in the ``airflow-ci-tfstate`` S3 bucket (``us-east-2``) with the
+``TerraformLocks`` DynamoDB table. Credentials/profile are taken from the environment
+(``AWS_PROFILE=...``), which also works for a future GitHub Actions plan/apply workflow
+assuming a role via OIDC — nothing is hardcoded in the provider.
-.. code-block:: bash
+.. note::
- cd eks
- tfenv install 1.5.3
+ The bucket also contains ``tf-state/eks.tfstate`` — the **2024 deployment's** state.
+ Its cluster is gone but leftovers (the ``airflow-vpc`` VPC, old node-group IAM roles,
+ the ``/aws/eks/airflow/cluster`` log group, the ``alias/eks/airflow`` KMS alias)
+ still exist in the account and are **deliberately left untouched**. The current
+ deployment therefore uses its own state key (``tf-state/eks-airflow-ci.tfstate``)
+ and the cluster name ``airflow-ci``, so every derived resource name avoids the 2024
+ ones. Do not import, modify, or delete the 2024 resources from this configuration.
+ For local iteration without touching the shared state, drop a
+ ``backend_override.tf`` (gitignored) with ``terraform { backend "local" {} }``
+ next to the sources.
Plan and deploy the infrastructure
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: bash
+ cd eks
+
# Initialize the Terraform modules and install the required providers,
# just the first time or when you add a new module
AWS_PROFILE=airflow terraform init
diff --git a/terraform/eks/.terraform.lock.hcl b/terraform/eks/.terraform.lock.hcl
index a3a0c54..b5d5d74 100644
--- a/terraform/eks/.terraform.lock.hcl
+++ b/terraform/eks/.terraform.lock.hcl
@@ -2,104 +2,109 @@
# Manual edits may be lost in future updates.
provider "registry.terraform.io/hashicorp/aws" {
- version = "5.64.0"
- constraints = ">= 4.33.0, >= 5.46.0, >= 5.61.0, ~> 5.64.0"
+ version = "6.56.0"
+ constraints = ">= 6.0.0, ~> 6.0, >= 6.28.0, >= 6.52.0"
hashes = [
- "h1:YH4I78rsS9t+YoGMPNzrM53aWi0Rb9Nud16iusrSXMg=",
- "zh:1d361f8062c68c9d5ac14b0aa8390709542129b8a9b258e61bbbabc706078b44",
- "zh:39dcbf53e3896bdd77071384c8fad4a5862c222c73f3bcf356aca488101f22fd",
- "zh:3fad63505f0c5b6f01cc9a6ef02b2226983b79424126a9caf6eb724f654299f4",
- "zh:53a8b90d00829cc27e3171a13a8ff1404ee0ea018e73f31d3f916d246cc39613",
- "zh:5734c25ef5a04b40f3c1ac5f817f11e42ee3328f74dbc141c0e64afbb0acc834",
- "zh:66ea14dbd87f291ce4a877123363933d3ca4022f209f885807a6689c22c24e80",
- "zh:68e79654ad0894a3d93134c3377748ace3058d5fad5ec09d1e9a8f8f9b8a47ea",
- "zh:7b74259d0ceef0c49cea6bcd171df997b6bad141085bbadded15b440faeb0eee",
- "zh:988ebfb5d115dc57070b5abf2e4200ad49cde535f27fd2ba5e34cf9ab336a57f",
+ "h1:iWz9BgFQaDPA1ChVGYvgI3MlUnx3wSQCA2ggYqDPcz8=",
+ "zh:2b3fbb3bebcc663b85d5fd9bbc2d131ab89322d696ff5c6ac6b7ffb7b5fe92e7",
+ "zh:30e56ccc7f33a7778ab323a28fe893d8e9200dc5fb92ccb7023bee808db3c1b0",
+ "zh:67dca271bef16547ef8ab5a6349f9bce39d91d7c1ae3d8388ada687ca774ba44",
+ "zh:824c812695b14d2fddad5e22339d4520e16d9c875f5d5095f29003f49a6fd124",
+ "zh:8372b12e30078d1df8b52e1285fd0d9d35160a7d18c3b0211f55229e5a832fd3",
+ "zh:8922b45ab65e272e8951f70d890444ddb3441170d8fa6f51298f24f20d21943d",
+ "zh:8fc4fb94ac8547717128cbcf296ac0ba0918738c4882029cbba46bd4812eb5e4",
"zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425",
- "zh:a0a2d4efe2835f0101a0a5024e044a3f28c00e10a8d87fce89c707ef6db75cea",
- "zh:aecb3e4b9121771dee9cac7975bf5d0657b5f3e8b57788c455beaeb0f3c48d93",
- "zh:d2d3393170b8ef761d3146f39f6788c4a3e876e6c5d4cedca4870c2680688ae6",
- "zh:daba5a005c1baa4a5eefbfb86d43ccf880eb5b42e8136f0d932f55886d72bda0",
- "zh:de16a6ff3baacdaf9609a0a89aa1913fc19cccaf5ee0fc1c49c5a075baa47c02",
+ "zh:a2efcd0174b79c1dffce48a5984f137ebc6f67d2c2ee966b47210e1faeb05cf2",
+ "zh:a4a50aa269a19c1d664b0dd59ee8047ed8a4a5b449f54733518309feccce1f43",
+ "zh:a830d80316b3c3127c9e0c77b9d4f50702c9d1a884c08973ac50b326d9ac734a",
+ "zh:d7e1b9bb2df3cdde8381ab101a807ae54e72c156d13a6b73dae2b922dca0c9f5",
+ "zh:d87c2a1cfc03b01cf13dff3700898ab990e1817326728824da8499dd0129da72",
+ "zh:fbb1848a597263c66dc6587ec8c3e0586e505e36f99d23752763534f62a3fef7",
+ "zh:fcb54d08eb3c763f01223e12b4eacbd018478e8e67c6b8611940564dfbea3d90",
+ "zh:ff6df70b2b2f75bd9000630f131c02e6bc2b9172cdb2547030f3fc019a3f3bc5",
]
}
provider "registry.terraform.io/hashicorp/cloudinit" {
- version = "2.3.4"
+ version = "2.4.0"
constraints = ">= 2.0.0"
hashes = [
- "h1:S3j8poSaLbaftlKq2STBkQEkZH253ZLaHhBHBifdpBQ=",
- "zh:09f1f1e1d232da96fbf9513b0fb5263bc2fe9bee85697aa15d40bb93835efbeb",
- "zh:381e74b90d7a038c3a8dcdcc2ce8c72d6b86da9f208a27f4b98cabe1a1032773",
- "zh:398eb321949e28c4c5f7c52e9b1f922a10d0b2b073b7db04cb69318d24ffc5a9",
- "zh:4a425679614a8f0fe440845828794e609b35af17db59134c4f9e56d61e979813",
- "zh:4d955d8608ece4984c9f1dacda2a59fdb4ea6b0243872f049b388181aab8c80a",
+ "h1:Bx3XQkBSY3RAGwLZb8hyi8AhvahPNlt4mlyZhW9guOI=",
+ "zh:1b0fe71b8e87a068f7cd9faaa733100ab72ab61ce812b8bd2b8e3e6ea3907b2d",
+ "zh:2aa9631ad64cfda1eb58f147619b631dadedfaf9453b422aa5ada2d3861183c1",
+ "zh:2c5f35463bdfb2f87d3576b81e62c30f8109e67bb6f21ffcbc46a855811455c0",
+ "zh:5970bcad151ea236bd262ada1a5a23bfbc1716f94a4e8b16ab2bcdda91d6a671",
"zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
- "zh:a48fbee1d58d55a1f4c92c2f38c83a37c8b2f2701ed1a3c926cefb0801fa446a",
- "zh:b748fe6631b16a1dafd35a09377c3bffa89552af584cf95f47568b6cd31fc241",
- "zh:d4b931f7a54603fa4692a2ec6e498b95464babd2be072bed5c7c2e140a280d99",
- "zh:f1c9337fcfe3a7be39d179eb7986c22a979cfb2c587c05f1b3b83064f41785c5",
- "zh:f58fc57edd1ee3250a28943cd84de3e4b744cdb52df0356a53403fc240240636",
- "zh:f5f50de0923ff530b03e1bca0ac697534d61bb3e5fc7f60e13becb62229097a9",
+ "zh:79a0676909732b6ec0441a733af6383513cde3bd2cef5c1ad0a74131e1286a04",
+ "zh:818f16141481a1202b3977becd19a12d4d46cd2e3f5753f5d0d0049adacf8f8c",
+ "zh:948d98716831087e69eca99f91ed7964cc537f3aca279f7494645ee56c9dc4ec",
+ "zh:a75e78889565a51df3e8e3af207e36e5ddb25e47ce1780a784c82dc3c3109b67",
+ "zh:a9c6e455d52b1bba5272bd87a35cfabcfd6d903dcbe42e2de926228dbb1e39b2",
+ "zh:b846805d8c2f5d1d6c2ffeeaf32109d9af7db7fa3c56929bfc1dcfaadf9c8bd8",
+ "zh:c3e5279756b46c4f49a6f4c81347fbe2fffebb2bf18a5c24664830304a1f6a8e",
+ "zh:c8be7b31893163d0046b0137a6100533f07e8efd192a1903b6bb4c42be12dceb",
]
}
provider "registry.terraform.io/hashicorp/null" {
- version = "3.2.2"
+ version = "3.3.0"
constraints = ">= 3.0.0"
hashes = [
- "h1:IMVAUHKoydFrlPrl9OzasDnw/8ntZFerCC9iXw1rXQY=",
- "zh:3248aae6a2198f3ec8394218d05bd5e42be59f43a3a7c0b71c66ec0df08b69e7",
- "zh:32b1aaa1c3013d33c245493f4a65465eab9436b454d250102729321a44c8ab9a",
- "zh:38eff7e470acb48f66380a73a5c7cdd76cc9b9c9ba9a7249c7991488abe22fe3",
- "zh:4c2f1faee67af104f5f9e711c4574ff4d298afaa8a420680b0cb55d7bbc65606",
- "zh:544b33b757c0b954dbb87db83a5ad921edd61f02f1dc86c6186a5ea86465b546",
- "zh:696cf785090e1e8cf1587499516b0494f47413b43cb99877ad97f5d0de3dc539",
- "zh:6e301f34757b5d265ae44467d95306d61bef5e41930be1365f5a8dcf80f59452",
+ "h1:l+dm3lhmu4ys7GbvIldfn544olSPH0DOiYruuFSfQkY=",
+ "zh:021748b5ea3b5f6956f2e75c42c5cdc113b391fb98ac71364a4965d23b37000f",
+ "zh:3b27956f8541d46704fda234e0d535c2ae2a4b33411848b1ee262a1ec03568b0",
+ "zh:3de4ed47d6d0f4d8edba4a5092c7c9799950eda63989d8d0d2586e6afcb0aa20",
+ "zh:57ed8935c7d56dbc91cf2673534582cacfaab7a2f105f51d9f797e99df0c0c47",
+ "zh:58e176ba1d142827089e30e0711e007309a9f2726e8881986da5026e9778fdf4",
+ "zh:5949c4a3d4a93f841f155cdb7e991c087e637145c1630572e21948224f8f4923",
+ "zh:76d60f366b743003c1b085afa769b45b2198ee919927e45807d7d44fb42c067d",
"zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
- "zh:913a929070c819e59e94bb37a2a253c228f83921136ff4a7aa1a178c7cce5422",
- "zh:aa9015926cd152425dbf86d1abdbc74bfe0e1ba3d26b3db35051d7b9ca9f72ae",
- "zh:bb04798b016e1e1d49bcc76d62c53b56c88c63d6f2dfe38821afef17c416a0e1",
- "zh:c23084e1b23577de22603cff752e59128d83cfecc2e6819edadd8cf7a10af11e",
+ "zh:79cd1bab1261a07f84e917191d7ddc4340ac5f5524283767256f7ffd7f87caf0",
+ "zh:8ec9083038cf710b30e319eaa467c9df7fa52bbd9969b61053a35bc2cdd2e0a6",
+ "zh:a6e502cb579685ab7aeb886c2bb11ddd9cfed74b41008592d57cbc3351a9218b",
+ "zh:acb74d6b4f66ff6acfcda315df802a7432170ef3955c9b432cb4580767004006",
+ "zh:f0ce55d8d9ffdb33dab612b1246f9bab060a9d54fc32ce2b4a038646155660af",
]
}
provider "registry.terraform.io/hashicorp/time" {
- version = "0.12.0"
+ version = "0.14.0"
constraints = ">= 0.9.0"
hashes = [
- "h1:Os2Ok7txtlUJHh6Hg7o+74Ql85SnRb/fGmah22yXpLw=",
- "zh:019a4c09af254ef80b72cf0d843dfe72d99483e227138cf5b514a1b9977ab4c3",
- "zh:0ae310ec740ebc6f275529507d60bb747d0bf39e72fc5a2fa90d74486006132c",
- "zh:13d6aec117f05237fbf8c7d91d6ebb19797b00aa87e7a812642d3ea4738a394e",
- "zh:2e87abbc261f9317d0c2ef26e01d5fabf77679da7d2cac6f47df7d198f720989",
- "zh:4a6d471176ce0264455aa7d5457b8702f78400010c201c1719708958a1b7b647",
+ "h1:4EThC3ocCFiFPMZQSUvSGSxoJqBcGWxMcFYmL67uS7Y=",
+ "zh:12abfd6b800e4d7fa6db7310dec8ffd440b31993861ef188c7ed5260b3073937",
+ "zh:23005521e800bb19e1597bf755c5f70d675d30b685d4255001ed5fa47d9df3f1",
+ "zh:2fea249b582ae97cd1cc10385187ea50993bb47c28cc5df0305e57ceaabf0a10",
+ "zh:322018d3b987b7aad08697178029a2bb667bed699e88328f0c89c52a2fd41341",
+ "zh:32a08e98fce2d273cb9b2c89d6c54727cc9f0a32e15bfd896be4e02cc6b48f95",
+ "zh:3db89aabd0e619616bd4b0f8b373a7586dfe60feffcea12a84a0bdbc445714b3",
+ "zh:7488f56c81d742dc020f29063626c8f07ca188aa97be61e7307e8d62397020a2",
"zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
- "zh:8978d5474a6da30bc0ad21c17db188d6918cacf3df3f6506b72ef3a268d53e2e",
- "zh:b109efe138dfcb45dc04a9cc6809d185ab8b0ebc12040847c2dac430fda5af68",
- "zh:b58e039b9106ac0a8de3c07f53b5279d7f0215fb35f2d23df642dfce0875382f",
- "zh:ba2cbb2e515922d13efe3a46647be84f5426fcfcaa0f1520b3efeab8db847ed3",
- "zh:c6c1ef1f26f25bca3abb5e07fa33dca37ed39cc26d0ff877964f2ffe5edd618c",
- "zh:f8e171f923b7d2e789abd034072465dec3e6133c3a7644b7a7a965a74d52224e",
+ "zh:7cb4067f2e7559b13f7562ef722f948950901eb37834873e98360ab28f66e9d7",
+ "zh:9d552c8345f61e1b7db8e725144981345f18ac1014d58d6f5ddf0928a195fffb",
+ "zh:a8e69fb6b97fc9d86fb19a9f4d42abe33c4a68e700b15387ce2e17d2b9934bed",
+ "zh:aeeb900eb8dd0f790c60ea5c0e0c8d42bd6e4a54f391681d4decca15b544394b",
+ "zh:c239c619101a8c95e1f14061eb973c57a8d15fa0e68878ced5bbd76858ee5b79",
]
}
provider "registry.terraform.io/hashicorp/tls" {
- version = "4.0.5"
- constraints = ">= 3.0.0"
+ version = "4.3.0"
+ constraints = ">= 4.0.0"
hashes = [
- "h1:zeG5RmggBZW/8JWIVrdaeSJa0OG62uFX5HY1eE8SjzY=",
- "zh:01cfb11cb74654c003f6d4e32bbef8f5969ee2856394a96d127da4949c65153e",
- "zh:0472ea1574026aa1e8ca82bb6df2c40cd0478e9336b7a8a64e652119a2fa4f32",
- "zh:1a8ddba2b1550c5d02003ea5d6cdda2eef6870ece86c5619f33edd699c9dc14b",
- "zh:1e3bb505c000adb12cdf60af5b08f0ed68bc3955b0d4d4a126db5ca4d429eb4a",
- "zh:6636401b2463c25e03e68a6b786acf91a311c78444b1dc4f97c539f9f78de22a",
- "zh:76858f9d8b460e7b2a338c477671d07286b0d287fd2d2e3214030ae8f61dd56e",
- "zh:a13b69fb43cb8746793b3069c4d897bb18f454290b496f19d03c3387d1c9a2dc",
- "zh:a90ca81bb9bb509063b736842250ecff0f886a91baae8de65c8430168001dad9",
- "zh:c4de401395936e41234f1956ebadbd2ed9f414e6908f27d578614aaa529870d4",
- "zh:c657e121af8fde19964482997f0de2d5173217274f6997e16389e7707ed8ece8",
- "zh:d68b07a67fbd604c38ec9733069fbf23441436fecf554de6c75c032f82e1ef19",
+ "h1:j/BqLS2N2AScZyotd9nZpHdieJ7e5S8y+A+ZfIu8kL8=",
+ "zh:0ab58d6f8991d436c7d2dbd89ed814709b949b07ac5a54ee53b0aec1fa772a8b",
+ "zh:60b347abcb56f45d97c56f14d895069cd15a83993f199777f571b79fea3642ee",
+ "zh:6889be32640349230de3f23856e6f04e0e9ced4a84a27d3f552fa54684448218",
+ "zh:73f8e1ecf7135033165fb14b7e8bf4d656f3ce13065ec35762ea0481975328c7",
+ "zh:94ce25ee253eca0b42cae9c856b36bca8103b6453012d1b279c3623c805f2d42",
+ "zh:96bc6de9fd67bc446fd11257872e1ffb1029a996ed1d65a3f6b43f6d408ad9ab",
+ "zh:97c609a310a51bfd504d704e036d72064a84bf0bdb36cc08cd4cc66098212b41",
+ "zh:a12c16e94533c5bd123f75032576b9dc91dd5d5ccd5f7cf331d0f2e1adc55cf8",
+ "zh:c4f014f876adf7af57188795050bda5b0029d8c7d7773031102b6c36dcf1fc21",
+ "zh:d9b0a21583aaa3df3a95394fb949a3c515ff71c2ff5a1fc4a73d364aa90bfca5",
+ "zh:da510d22f0c6d71ad19a76406f106b782448f512375787ecfabb338ed1e311a7",
+ "zh:f0e9447a9ce3a24cdaa113089e65663c836d8b9bfdb915a1c0284e0112cab5c0",
"zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
]
}
diff --git a/terraform/eks/asg_tags.tf b/terraform/eks/asg_tags.tf
deleted file mode 100644
index 3c05429..0000000
--- a/terraform/eks/asg_tags.tf
+++ /dev/null
@@ -1,67 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-######################################################################################################
-# In order to scale the node groups from zero, we need to add the labels and taints to the ASGs tags:
-# k8s.io/cluster-autoscaler/node-template/label/<label-key> = <label-value>
-# k8s.io/cluster-autoscaler/node-template/taint/<taint-key> = <taint-value>:<taint-effect>
-######################################################################################################
-
-locals {
-
- taint_effects = {
- NO_SCHEDULE = "NoSchedule"
- NO_EXECUTE = "NoExecute"
- PREFER_NO_SCHEDULE = "PreferNoSchedule"
- }
-
- cluster_autoscaler_label_tags = merge([
- for name, group in module.eks.eks_managed_node_groups : {
- for label_name, label_value in coalesce(group.node_group_labels, {}) : "${name}|label|${label_name}" => {
- autoscaling_group = group.node_group_autoscaling_group_names[0],
- key = "k8s.io/cluster-autoscaler/node-template/label/${label_name}",
- value = label_value,
- }
- }
- ]...)
-
- cluster_autoscaler_taint_tags = merge([
- for name, group in module.eks.eks_managed_node_groups : {
- for taint in coalesce(group.node_group_taints, []) : "${name}|taint|${taint.key}" => {
- autoscaling_group = group.node_group_autoscaling_group_names[0],
- key = "k8s.io/cluster-autoscaler/node-template/taint/${taint.key}"
- value = "${taint.value}:${local.taint_effects[taint.effect]}"
- }
- }
- ]...)
-
- cluster_autoscaler_asg_tags = merge(local.cluster_autoscaler_label_tags, local.cluster_autoscaler_taint_tags)
-}
-
-resource "aws_autoscaling_group_tag" "cluster_autoscaler_label_tags" {
- for_each = local.cluster_autoscaler_asg_tags
-
- autoscaling_group_name = each.value.autoscaling_group
-
- tag {
- key = each.value.key
- value = each.value.value
-
- propagate_at_launch = true
- }
-}
-
diff --git a/terraform/eks/eks.tf b/terraform/eks/eks.tf
index 5b4203a..bd901db 100644
--- a/terraform/eks/eks.tf
+++ b/terraform/eks/eks.tf
@@ -18,218 +18,59 @@
data "aws_availability_zones" "available" {}
locals {
- cluster_name = "airflow"
+ cluster_name = "airflow-ci"
}
module "eks" {
source = "terraform-aws-modules/eks/aws"
- version = "20.24.0"
+ version = "~> 21.24"
- cluster_name = local.cluster_name
- cluster_version = "1.30"
+ name = local.cluster_name
+ kubernetes_version = var.kubernetes_version
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.public_subnets
- cluster_endpoint_public_access = true
-
- eks_managed_node_group_defaults = {
- ami_type = "AL2_ARM_64"
- use_custom_launch_template = false
- disk_size = 50
- desired_size = 0
- }
-
- eks_managed_node_groups = {
-
- default_nodes = {
- name = "default"
-
- instance_types = [
- "t4g.medium",
- "t4g.large",
- "t4g.xlarge",
- "m7g.medium",
- "m7g.large",
- "m7g.xlarge",
- ]
-
- min_size = 0
- max_size = 5
- desired_size = 1
-
- capacity_type = "SPOT"
-
- labels = {
- "node-type" = "default"
- }
+ endpoint_public_access = true
+ enable_cluster_creator_admin_permissions = true
+
+ addons = {
+ coredns = {}
+ kube-proxy = {}
+ # before_compute: the module disables EKS's bootstrap self-managed CNI when
+ # managed addons are used, so vpc-cni must exist BEFORE nodes boot — nodes
+ # without a CNI never become Ready and the node group lands in CREATE_FAILED
+ # (bitten by this on 2026-07-25 when a crashed apply skipped the addons).
+ vpc-cni = {
+ before_compute = true
}
-
- # GHA arm runners' nodes
- GHA_runners_small = {
- name = "gha-runners-small"
-
- instance_types = [for node_type in var.runners_node_types: "${node_type}.${var.small_runners_node_size}"]
-
- min_size = 0
- max_size = 30
-
- capacity_type = "SPOT"
-
- disk_size = 50
-
- labels = {
- "node-type" = "gha-runners"
- "size" = "small"
- "arch" = "arm64"
- }
-
- taints = [
- {
- "key" = "node-type"
- "value" = "gha-runners"
- "effect" = "NO_SCHEDULE"
- }
- ]
+ # Required for Karpenter's Pod Identity association.
+ eks-pod-identity-agent = {
+ before_compute = true
}
+ }
- GHA_runners_medium = {
- name = "gha-runners-medium"
-
- instance_types = [for node_type in var.runners_node_types: "${node_type}.${var.medium_runners_node_size}"]
-
- min_size = 0
- max_size = 30
-
- capacity_type = "SPOT"
-
- disk_size = 50
-
- labels = {
- "node-type" = "gha-runners"
- "size" = "medium"
- "arch" = "arm64"
- }
-
- taints = [
- {
- "key" = "node-type"
- "value" = "gha-runners"
- "effect" = "NO_SCHEDULE"
- }
- ]
- }
-
- GHA_runners_large = {
- name = "gha-runners-large"
-
- instance_types = [for node_type in var.runners_node_types: "${node_type}.${var.large_runners_node_size}"]
-
- min_size = 0
- max_size = 30
-
- capacity_type = "SPOT"
-
- labels = {
- "node-type" = "gha-runners"
- "size" = "large"
- "arch" = "arm64"
- }
-
- taints = [
- {
- "key" = "node-type"
- "value" = "gha-runners"
- "effect" = "NO_SCHEDULE"
- }
- ]
- }
-
- # GHA x64 runners' nodes
- x64_GHA_runners_small = {
- name = "gha-x64-runners-small"
-
- ami_type = "AL2_x86_64"
-
- instance_types = [for node_type in var.x64_runners_node_types: "${node_type}.${var.small_runners_x64_node_size}"]
-
- min_size = 0
- max_size = 30
-
- capacity_type = "SPOT"
-
- disk_size = 50
-
- labels = {
- "node-type" = "gha-runners"
- "size" = "small"
- "arch" = "x64"
- }
-
- taints = [
- {
- "key" = "node-type"
- "value" = "gha-runners"
- "effect" = "NO_SCHEDULE"
- }
- ]
- }
-
- x64_GHA_runners_medium = {
- name = "gha-x64-runners-medium"
-
- ami_type = "AL2_x86_64"
-
- instance_types = [for node_type in var.x64_runners_node_types: "${node_type}.${var.medium_runners_node_size}"]
-
- min_size = 0
- max_size = 30
-
- capacity_type = "SPOT"
-
- disk_size = 50
-
- labels = {
- "node-type" = "gha-runners"
- "size" = "medium"
- "arch" = "x64"
- }
-
- taints = [
- {
- "key" = "node-type"
- "value" = "gha-runners"
- "effect" = "NO_SCHEDULE"
- }
- ]
- }
-
- x64_GHA_runners_large = {
- name = "gha-x64-runners-large"
-
- ami_type = "AL2_x86_64"
-
- instance_types = [for node_type in var.x64_runners_node_types: "${node_type}.${var.large_runners_node_size}"]
-
- min_size = 0
- max_size = 30
+ # Karpenter's EC2NodeClass discovers the node security group by this tag.
+ node_security_group_tags = {
+ "karpenter.sh/discovery" = local.cluster_name
+ }
- capacity_type = "SPOT"
+ eks_managed_node_groups = {
+ # Small always-on group for the controllers (Karpenter, ARC, listeners).
+ # Everything else runs on Karpenter-provisioned nodes.
+ system = {
+ name = "airflow-ci-system"
+ ami_type = "AL2023_ARM_64_STANDARD"
+ instance_types = var.system_node_instance_types
+ capacity_type = "ON_DEMAND"
+
+ min_size = 2
+ max_size = 3
+ desired_size = 2
labels = {
- "node-type" = "gha-runners"
- "size" = "large"
- "arch" = "x64"
+ "airflow-ci/role" = "system"
}
-
- taints = [
- {
- "key" = "node-type"
- "value" = "gha-runners"
- "effect" = "NO_SCHEDULE"
- }
- ]
}
}
}
-
diff --git a/terraform/eks/iam.tf b/terraform/eks/iam.tf
deleted file mode 100644
index e2c3000..0000000
--- a/terraform/eks/iam.tf
+++ /dev/null
@@ -1,134 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-data "aws_iam_policy_document" "autoscaler_policy_document" {
- statement {
- effect = "Allow"
- actions = [
- "autoscaling:DescribeAutoScalingGroups",
- "autoscaling:DescribeAutoScalingInstances",
- "autoscaling:DescribeLaunchConfigurations",
- "autoscaling:DescribeScalingActivities",
- "autoscaling:DescribeTags",
- "ec2:DescribeInstanceTypes",
- "ec2:DescribeLaunchTemplateVersions"
- ]
- resources = ["*"]
- }
-
- statement {
- effect = "Allow"
- actions = [
- "autoscaling:SetDesiredCapacity",
- "autoscaling:TerminateInstanceInAutoScalingGroup",
- "ec2:DescribeImages",
- "ec2:GetInstanceTypesFromInstanceRequirements",
- "eks:DescribeNodegroup"
- ]
- resources = ["*"]
- }
-}
-
-resource "aws_iam_policy" "autoscaler_policy" {
- name = "eks-autoscaler-policy"
- description = "EKS Autoscaler Policy"
- policy = data.aws_iam_policy_document.autoscaler_policy_document.json
-}
-
-data "aws_iam_policy_document" "autoscaler_role_assume_policy_document" {
- statement {
- actions = [
- "sts:AssumeRoleWithWebIdentity"
- ]
-
- principals {
- type = "Federated"
- identifiers = [
- module.eks.oidc_provider_arn
- ]
- }
-
- condition {
- test = "StringEquals"
- variable = "${module.eks.oidc_provider}:sub"
- values = [
- "system:serviceaccount:infra:cluster-autoscaler"
- ]
- }
-
- condition {
- test = "StringEquals"
- variable = "${module.eks.oidc_provider}:aud"
- values = [
- "sts.amazonaws.com"
- ]
- }
-
- }
-}
-
-resource "aws_iam_role" "autoscaler_role" {
- name = "eks-autoscaler-role"
- assume_role_policy = data.aws_iam_policy_document.autoscaler_role_assume_policy_document.json
-}
-
-resource "aws_iam_role_policy_attachment" "autoscaler_policy_attachment" {
- role = aws_iam_role.autoscaler_role.name
- policy_arn = aws_iam_policy.autoscaler_policy.arn
-}
-
-data "aws_iam_policy_document" "ebs_csi_driver_assume_policy_document" {
- statement {
- actions = [
- "sts:AssumeRoleWithWebIdentity"
- ]
-
- principals {
- type = "Federated"
- identifiers = [
- module.eks.oidc_provider_arn
- ]
- }
-
- condition {
- test = "StringEquals"
- variable = "${module.eks.oidc_provider}:sub"
- values = [
- "system:serviceaccount:kube-system:ebs-csi-controller-sa"
- ]
- }
-
- condition {
- test = "StringEquals"
- variable = "${module.eks.oidc_provider}:aud"
- values = [
- "sts.amazonaws.com"
- ]
- }
-
- }
-}
-
-resource "aws_iam_role" "ebs_csi_driver" {
- name = "ebs-csi-driver"
- assume_role_policy = data.aws_iam_policy_document.ebs_csi_driver_assume_policy_document.json
-}
-
-resource "aws_iam_role_policy_attachment" "ebs_csi_driver" {
- role = aws_iam_role.ebs_csi_driver.name
- policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
-}
diff --git a/terraform/eks/vpc.tf b/terraform/eks/karpenter.tf
similarity index 50%
copy from terraform/eks/vpc.tf
copy to terraform/eks/karpenter.tf
index 1d80025..c334d69 100644
--- a/terraform/eks/vpc.tf
+++ b/terraform/eks/karpenter.tf
@@ -15,30 +15,22 @@
# specific language governing permissions and limitations
# under the License.
-module "vpc" {
- source = "terraform-aws-modules/vpc/aws"
- version = "5.13.0"
+# IAM + SQS plumbing for Karpenter (controller role via EKS Pod Identity, node
+# role + access entry, spot-interruption queue). Karpenter itself is installed
+# with helmfile (helm/infra-helmfile.yaml); the names below are referenced from
+# the helm values, so they are pinned instead of name-prefixed.
+module "karpenter" {
+ source = "terraform-aws-modules/eks/aws//modules/karpenter"
+ version = "~> 21.24"
- name = "airflow-vpc"
+ cluster_name = module.eks.cluster_name
- cidr = "10.1.0.0/16"
- azs = slice(data.aws_availability_zones.available.names, 0, 3)
+ # Referenced by helm/values/karpenter.yaml (settings.interruptionQueue).
+ queue_name = "airflow-ci-karpenter"
- private_subnets = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"]
- public_subnets = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"]
+ # Referenced by the EC2NodeClass (spec.role) in helm/karpenter-resources.
+ node_iam_role_name = "airflow-ci-runner-node"
+ node_iam_role_use_name_prefix = false
- enable_nat_gateway = false
- enable_dns_hostnames = true
-
- map_public_ip_on_launch = true
-
- public_subnet_tags = {
- "kubernetes.io/cluster/${local.cluster_name}" = "shared"
- "kubernetes.io/role/elb" = 1
- }
-
- private_subnet_tags = {
- "kubernetes.io/cluster/${local.cluster_name}" = "shared"
- "kubernetes.io/role/internal-elb" = 1
- }
-}
\ No newline at end of file
+ create_pod_identity_association = true
+}
diff --git a/terraform/eks/vpc.tf b/terraform/eks/outputs.tf
similarity index 50%
copy from terraform/eks/vpc.tf
copy to terraform/eks/outputs.tf
index 1d80025..f7d224d 100644
--- a/terraform/eks/vpc.tf
+++ b/terraform/eks/outputs.tf
@@ -15,30 +15,27 @@
# specific language governing permissions and limitations
# under the License.
-module "vpc" {
- source = "terraform-aws-modules/vpc/aws"
- version = "5.13.0"
-
- name = "airflow-vpc"
-
- cidr = "10.1.0.0/16"
- azs = slice(data.aws_availability_zones.available.names, 0, 3)
-
- private_subnets = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"]
- public_subnets = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"]
-
- enable_nat_gateway = false
- enable_dns_hostnames = true
-
- map_public_ip_on_launch = true
-
- public_subnet_tags = {
- "kubernetes.io/cluster/${local.cluster_name}" = "shared"
- "kubernetes.io/role/elb" = 1
- }
-
- private_subnet_tags = {
- "kubernetes.io/cluster/${local.cluster_name}" = "shared"
- "kubernetes.io/role/internal-elb" = 1
- }
-}
\ No newline at end of file
+output "cluster_name" {
+ description = "EKS cluster name"
+ value = module.eks.cluster_name
+}
+
+output "cluster_endpoint" {
+ description = "EKS API endpoint"
+ value = module.eks.cluster_endpoint
+}
+
+output "karpenter_node_iam_role_name" {
+ description = "IAM role assumed by Karpenter-launched nodes (EC2NodeClass spec.role)"
+ value = module.karpenter.node_iam_role_name
+}
+
+output "karpenter_interruption_queue" {
+ description = "SQS queue for spot interruption handling (karpenter settings.interruptionQueue)"
+ value = module.karpenter.queue_name
+}
+
+output "update_kubeconfig" {
+ description = "Command to configure kubectl"
+ value = "aws eks --region ${var.region} update-kubeconfig --name ${module.eks.cluster_name} --alias ${module.eks.cluster_name}"
+}
diff --git a/terraform/eks/provider.tf b/terraform/eks/provider.tf
index 89656fe..5388c88 100644
--- a/terraform/eks/provider.tf
+++ b/terraform/eks/provider.tf
@@ -15,7 +15,16 @@
# specific language governing permissions and limitations
# under the License.
+# Credentials come from the environment (AWS_PROFILE locally, OIDC role in a
+# future GitHub Actions workflow) — do not hardcode a profile here.
provider "aws" {
- profile = "airflow"
- region = var.region
+ region = var.region
+
+ default_tags {
+ tags = {
+ Project = "airflow-ci"
+ ManagedBy = "terraform"
+ Repo = "apache/airflow-ci-infra"
+ }
+ }
}
\ No newline at end of file
diff --git a/terraform/eks/runner_pod_identity.tf b/terraform/eks/runner_pod_identity.tf
new file mode 100644
index 0000000..bbd267d
--- /dev/null
+++ b/terraform/eks/runner_pod_identity.tf
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The airflow CI workflows pass artifacts between jobs via the S3-backed stash
+# actions (.github/actions/stash-s3-*). Runner pods get scoped S3 access
+# through EKS Pod Identity — node IAM credentials are deliberately
+# unreachable from pods (IMDS hop limit 1 on the EC2NodeClass).
+locals {
+ # Same-region successor of the us-east-1 `airflow-ci-runner-cache` bucket
+ # (bucket names are global, so the regional suffix disambiguates). Contents
+ # are disposable inter-job artifacts — nothing was migrated.
+ stash_bucket = "airflow-ci-runner-cache-us-east-2"
+
+ # One service account per runner scale set, created by the gha-runner-scale-set
+ # chart as "<scale-set-name>-gha-rs-no-permission" in the arc-runners namespace.
+ runner_scale_set_names = [
+ "airflow-runner-amd-small",
+ "airflow-runner-amd-medium",
+ "airflow-runner-amd-xl",
+ "airflow-runner-arm-small",
+ "airflow-runner-arm-medium",
+ "airflow-runner-arm-xl",
+ ]
+}
+
+resource "aws_s3_bucket" "stash" {
+ bucket = local.stash_bucket
+}
+
+resource "aws_s3_bucket_public_access_block" "stash" {
+ bucket = aws_s3_bucket.stash.id
+
+ block_public_acls = true
+ block_public_policy = true
+ ignore_public_acls = true
+ restrict_public_buckets = true
+}
+
+# Stashes are short-lived hand-offs between jobs of a run; the actions ask for
+# ~2-day retention, the lifecycle rule is the backstop that actually deletes.
+resource "aws_s3_bucket_lifecycle_configuration" "stash" {
+ bucket = aws_s3_bucket.stash.id
+
+ rule {
+ id = "expire-stashes"
+ status = "Enabled"
+
+ filter {}
+
+ expiration {
+ days = 3
+ }
+
+ abort_incomplete_multipart_upload {
+ days_after_initiation = 1
+ }
+ }
+}
+
+data "aws_iam_policy_document" "runner_pods_assume" {
+ statement {
+ actions = ["sts:AssumeRole", "sts:TagSession"]
+ principals {
+ type = "Service"
+ identifiers = ["pods.eks.amazonaws.com"]
+ }
+ }
+}
+
+data "aws_iam_policy_document" "runner_pods_stash" {
+ statement {
+ actions = ["s3:GetObject", "s3:PutObject"]
+ resources = ["arn:aws:s3:::${local.stash_bucket}/*"]
+ }
+ statement {
+ actions = ["s3:ListBucket", "s3:GetBucketLocation"]
+ resources = ["arn:aws:s3:::${local.stash_bucket}"]
+ }
+}
+
+resource "aws_iam_role" "runner_pods" {
+ name = "airflow-ci-runner-pods"
+ assume_role_policy = data.aws_iam_policy_document.runner_pods_assume.json
+}
+
+resource "aws_iam_role_policy" "runner_pods_stash" {
+ name = "stash-bucket-access"
+ role = aws_iam_role.runner_pods.id
+ policy = data.aws_iam_policy_document.runner_pods_stash.json
+}
+
+resource "aws_eks_pod_identity_association" "runner_pods" {
+ for_each = toset(local.runner_scale_set_names)
+
+ cluster_name = module.eks.cluster_name
+ namespace = "arc-runners"
+ service_account = "${each.value}-gha-rs-no-permission"
+ role_arn = aws_iam_role.runner_pods.arn
+}
diff --git a/terraform/eks/state.tf b/terraform/eks/state.tf
index 025e912..8801ae2 100644
--- a/terraform/eks/state.tf
+++ b/terraform/eks/state.tf
@@ -18,10 +18,16 @@
terraform {
backend "s3" {
- region = "us-east-2"
- bucket = "airflow-ci-tfstate"
- key = "tf-state/eks.tfstate"
- encrypt = true
- dynamodb_table = "TerraformLocks"
+ region = "us-east-2"
+ bucket = "airflow-ci-tfstate"
+ # Fresh key for the 2026 airflow-ci cluster. tf-state/eks.tfstate is the
+ # 2024 deployment's state and is deliberately left untouched (its cluster
+ # is gone but the VPC/IAM leftovers still exist and must not be disturbed).
+ key = "tf-state/eks-airflow-ci.tfstate"
+ encrypt = true
+ # S3-native locking (.tflock object next to the state). Replaces the
+ # deprecated dynamodb_table locking; the TerraformLocks DynamoDB table is
+ # still referenced by the 2024 state and must not be deleted.
+ use_lockfile = true
}
}
diff --git a/terraform/eks/variables.tf b/terraform/eks/variables.tf
index 045c496..deeb5a0 100644
--- a/terraform/eks/variables.tf
+++ b/terraform/eks/variables.tf
@@ -21,48 +21,14 @@ variable "region" {
default = "us-east-2"
}
-variable "runners_node_types" {
- description = "Node type for the runners"
- type = list(string)
- default = [
- "t4g",
- "m7g",
- "m6g",
- ]
-}
-
-variable "x64_runners_node_types" {
- description = "X64 node type for the runners"
- type = list(string)
- default = [
- "m7a",
- "m6a",
- "t3",
- "t3a",
- ]
-}
-
-variable "small_runners_node_size" {
- description = "Node size for the small runners"
- type = string
- default = "medium"
-}
-
-variable "small_runners_x64_node_size" {
- # This is a separate variable because the r series doesn't have a medium size
- description = "Node size for the small runners"
+variable "kubernetes_version" {
+ description = "EKS Kubernetes version; keep within standard support to avoid extended-support fees"
type = string
- default = "large"
+ default = "1.34"
}
-variable "medium_runners_node_size" {
- description = "Node size for the medium runners"
- type = string
- default = "xlarge"
+variable "system_node_instance_types" {
+ description = "Instance types for the small always-on system node group (Karpenter + ARC controllers)"
+ type = list(string)
+ default = ["t4g.medium"]
}
-
-variable "large_runners_node_size" {
- description = "Node size for the large runners"
- type = string
- default = "2xlarge"
-}
\ No newline at end of file
diff --git a/terraform/eks/version.tf b/terraform/eks/version.tf
index a215415..93977fd 100644
--- a/terraform/eks/version.tf
+++ b/terraform/eks/version.tf
@@ -19,8 +19,9 @@ terraform {
required_providers {
aws = {
source = "hashicorp/aws"
- version = "~> 5.64.0"
+ version = "~> 6.0"
}
}
- required_version = "~> 1.9.5"
+ # 1.10+ for S3-native state locking (use_lockfile)
+ required_version = ">= 1.10.0"
}
\ No newline at end of file
diff --git a/terraform/eks/vpc.tf b/terraform/eks/vpc.tf
index 1d80025..3ff2339 100644
--- a/terraform/eks/vpc.tf
+++ b/terraform/eks/vpc.tf
@@ -15,17 +15,21 @@
# specific language governing permissions and limitations
# under the License.
+# Nodes live in public subnets with public IPs on purpose: a NAT gateway costs
+# ~$400/yr while public IPv4 on ephemeral runner nodes is a few $/mo, and the
+# security groups allow no inbound traffic (runners are outbound-only).
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
- version = "5.13.0"
+ version = "~> 6.6"
- name = "airflow-vpc"
+ name = "airflow-ci-vpc"
cidr = "10.1.0.0/16"
azs = slice(data.aws_availability_zones.available.names, 0, 3)
- private_subnets = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"]
- public_subnets = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"]
+ # Docker inside the runners carves networks out of 172.16.0.0/14 — the VPC
+ # range must stay clear of it, hence 10.x.
+ public_subnets = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"]
enable_nat_gateway = false
enable_dns_hostnames = true
@@ -34,11 +38,7 @@ module "vpc" {
public_subnet_tags = {
"kubernetes.io/cluster/${local.cluster_name}" = "shared"
- "kubernetes.io/role/elb" = 1
+ # Karpenter's EC2NodeClass discovers subnets by this tag.
+ "karpenter.sh/discovery" = local.cluster_name
}
-
- private_subnet_tags = {
- "kubernetes.io/cluster/${local.cluster_name}" = "shared"
- "kubernetes.io/role/internal-elb" = 1
- }
-}
\ No newline at end of file
+}