You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "shahar1 (via GitHub)" <gi...@apache.org> on 2026/09/25 13:51:17 UTC
[PR] Add Terraform for the RunsOn stack of the apache organization [airflow-ci-infra]
shahar1 opened a new pull request, #65:
URL: https://github.com/apache/airflow-ci-infra/pull/65
[AIP-118](https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974652) selected [RunsOn](https://runs-on.com/) for Airflow's self-hosted GitHub Actions runners. This adds `terraform/runs-on/`, a root module that creates RunsOn's CloudFormation stack for the **apache** organization from the vendor's version-pinned template (`v3.3.2`), with the parameters validated on a fork on 2026-09-25 (full `Tests (AMD)` graph green apart from two non-hermetic tests, details in the [AIP-118 appendix](https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974672)).
What is in the module
- `aws_cloudformation_stack.runs_on` — stack `runs-on`, `GithubOrganization=apache`, VPC `10.2.0.0/16`, `prevent_destroy`, own S3 state key `tf-state/runs-on.tfstate`.
- License key handling: RunsOn accepts `LicenseKey` as the ARN of an SSM SecureString that must be named `/<stack>/license-key`. The parameter is created by hand (README step 1); Terraform reads it **without decryption** only to prove it exists and passes the ARN, so the key never appears in variables, plan output or state. Secrets Manager is not supported by the template.
- `runs-on.yml.example` — runner presets for the three size classes (small 4 vCPU/16 GB, medium 8/32, xl 32/128, AMD and ARM), to be committed in `apache/.github-private`.
- README with the manual steps (SSM parameter, GitHub App registration via the stack's entry point — needs an org owner / ASF Infra), upgrade/rollback and what was learned in the validation.
The fork validation stack `runs-on-shahar1` is deliberately not managed here and must never be imported.
Status: applied on 2026-09-25 with a local backend — stack `runs-on` is `CREATE_COMPLETE`, license validated by the control plane. Pending: GitHub App registration for the `apache` organization (ASF Infra), then presets and workflow routing in `apache/airflow`. A GitHub Actions workflow to run this Terraform (OIDC role, S3 state) is a follow-up, not part of this PR.
Checks run: `terraform fmt -check`, `terraform validate`, `terraform plan` (1 to add) and `apply` against the account.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Fable 5.1)
Generated-by: Claude Code (Fable 5.1) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Terraform for the RunsOn stack of the apache organization [airflow-ci-infra]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #65:
URL: https://github.com/apache/airflow-ci-infra/pull/65#discussion_r4359708941
##########
terraform/runs-on/README.rst:
##########
@@ -0,0 +1,165 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+RunsOn — self-hosted CI runners on AWS
+======================================
+
+`RunsOn <https://runs-on.com/>`_ is the approach selected by
+`AIP-118 <https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974652>`_ for Airflow's
+self-hosted GitHub Actions runners. It packages runner provisioning into a CloudFormation stack in
+Airflow's AWS account: a small control plane (ECS Fargate + API Gateway) receives GitHub's
+``workflow_job`` webhooks and launches one ephemeral EC2 instance per job, Spot first, sized from
+the job's ``runs-on:`` label. RunsOn's `free open-source license <https://runs-on.com/billing/free/>`_
+covers the product; EC2/EBS/S3 usage is billed to the account.
+
+This module creates the stack for the **apache** organization from RunsOn's published,
+version-pinned template, with the configuration validated on 2026-09-25 (full ``Tests (AMD)``
+graph green apart from two non-hermetic tests — see the AIP-118 appendix).
+
+Files
+-----
+
+``main.tf``
+ The ``aws_cloudformation_stack`` resource, its parameters and the license-key lookup.
+``variables.tf``
+ All variables have defaults: ``stack_name`` (``runs-on``), ``github_organization``
+ (``apache``), ``alert_email`` (``shahar@apache.org``), ``runs_on_version`` (``v3.3.2``),
+ ``vpc_cidr`` (``10.2.0.0/16``) and ``region`` (``us-east-2``).
+``state.tf``
+ S3 backend ``airflow-ci-tfstate`` / ``tf-state/runs-on.tfstate`` (its own key next to the
+ EKS states).
+``runs-on.yml.example``
+ Runner presets for Airflow's three size classes, to be committed as ``.github/runs-on.yml``
+ in the organization's ``.github-private`` repository (or a repository's default branch).
+
+Step 1 — create the license-key parameter (manual, once)
+--------------------------------------------------------
+
+RunsOn accepts ``LicenseKey`` either as plain text or as the ARN of an SSM Parameter Store
+parameter that must be named ``/<stack name>/license-key`` (`reference
+<https://runs-on.com/docs/maintenance/configuration/flex/cloudformation/>`_). The ARN form keeps
+the key out of Terraform variables, plan output and state: ``main.tf`` reads the parameter
+**without decryption** (only to prove it exists) and passes its ARN; the stack's service role is
+the only principal that decrypts it. Secrets Manager is not supported by the template.
+
+.. code-block:: bash
+
+ aws ssm put-parameter --region us-east-2 \
+ --name /runs-on/license-key \
Review Comment:
Dropped the variable in 87e1da6. `us-east-2` is now written out in `provider.tf`, with a comment saying the backend in `state.tf` and the README commands must change with it. The README's variables list says the same.
---
Drafted-by: Claude Code (Opus 5.5) (no human review before posting)
##########
terraform/runs-on/outputs.tf:
##########
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+output "entry_point" {
+ description = "RunsOn web entry point: open it once after creation to register the GitHub App."
+ value = aws_cloudformation_stack.runs_on.outputs["RunsOnEntryPoint"]
+}
+
+output "cache_bucket" {
+ description = "S3 bucket the runners can use under the cache/ prefix (stash payloads)."
+ value = aws_cloudformation_stack.runs_on.outputs["RunsOnBucketCache"]
+}
+
+output "app_version" {
Review Comment:
Added in 87e1da6.
---
Drafted-by: Claude Code (Opus 5.5) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Terraform for the RunsOn stack of the apache organization [airflow-ci-infra]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #65:
URL: https://github.com/apache/airflow-ci-infra/pull/65#discussion_r4359706844
##########
terraform/runs-on/main.tf:
##########
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+data "aws_partition" "current" {}
+data "aws_caller_identity" "current" {}
+
+# The license key is never in Terraform state: RunsOn accepts LicenseKey as the ARN of an SSM
+# SecureString parameter that MUST be named /<stack name>/license-key, and only the stack's
+# service role decrypts it. The parameter is created by hand (see README.rst); this lookup
+# without decryption only proves it exists before the stack is planned, storing the ciphertext.
+data "aws_ssm_parameter" "license_key" {
+ name = "/${var.stack_name}/license-key"
+ with_decryption = false
+}
+
+resource "aws_cloudformation_stack" "runs_on" {
+ name = var.stack_name
+ template_url = "https://runs-on.s3.eu-west-1.amazonaws.com/cloudformation/template-${var.runs_on_version}.yaml"
+ capabilities = ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM", "CAPABILITY_AUTO_EXPAND"]
+
+ # Everything not listed keeps the template default; the values match the stack validated on
+ # 2026-09-25 (see the AIP-118 appendix) apart from the organization and the CIDR.
+ parameters = {
+ GithubOrganization = var.github_organization
+ LicenseKey = data.aws_ssm_parameter.license_key.arn
+ EmailAddress = var.alert_email
+ Environment = "production"
+ VpcCidrBlock = var.vpc_cidr
+ Private = "false" # runners get public IPs: no NAT cost, Docker Hub pulls per runner IP
+ SSHAllowed = "false"
+ SSMAllowed = "true"
Review Comment:
Done in 87e1da6: new `enable_admin_routes` variable, default `false`, passed to `EnableAdminRoutes`. The README's GitHub App step now says to apply with `-var enable_admin_routes=true` only for the (re-)registration and to apply again without it.
Applied to the `apache` stack: CloudFormation removed the setup Lambda and the `/`, `/setup` and `/readyz` routes, leaving only `/github/webhooks`. The app was already registered, so nothing was lost.
---
Drafted-by: Claude Code (Opus 5.5) (no human review before posting)
##########
terraform/runs-on/README.rst:
##########
@@ -0,0 +1,165 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+RunsOn — self-hosted CI runners on AWS
+======================================
+
+`RunsOn <https://runs-on.com/>`_ is the approach selected by
+`AIP-118 <https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974652>`_ for Airflow's
+self-hosted GitHub Actions runners. It packages runner provisioning into a CloudFormation stack in
+Airflow's AWS account: a small control plane (ECS Fargate + API Gateway) receives GitHub's
+``workflow_job`` webhooks and launches one ephemeral EC2 instance per job, Spot first, sized from
+the job's ``runs-on:`` label. RunsOn's `free open-source license <https://runs-on.com/billing/free/>`_
+covers the product; EC2/EBS/S3 usage is billed to the account.
+
+This module creates the stack for the **apache** organization from RunsOn's published,
+version-pinned template, with the configuration validated on 2026-09-25 (full ``Tests (AMD)``
+graph green apart from two non-hermetic tests — see the AIP-118 appendix).
+
+Files
+-----
+
+``main.tf``
+ The ``aws_cloudformation_stack`` resource, its parameters and the license-key lookup.
+``variables.tf``
+ All variables have defaults: ``stack_name`` (``runs-on``), ``github_organization``
+ (``apache``), ``alert_email`` (``shahar@apache.org``), ``runs_on_version`` (``v3.3.2``),
+ ``vpc_cidr`` (``10.2.0.0/16``) and ``region`` (``us-east-2``).
+``state.tf``
+ S3 backend ``airflow-ci-tfstate`` / ``tf-state/runs-on.tfstate`` (its own key next to the
+ EKS states).
+``runs-on.yml.example``
+ Runner presets for Airflow's three size classes, to be committed as ``.github/runs-on.yml``
+ in the organization's ``.github-private`` repository (or a repository's default branch).
+
+Step 1 — create the license-key parameter (manual, once)
+--------------------------------------------------------
+
+RunsOn accepts ``LicenseKey`` either as plain text or as the ARN of an SSM Parameter Store
+parameter that must be named ``/<stack name>/license-key`` (`reference
+<https://runs-on.com/docs/maintenance/configuration/flex/cloudformation/>`_). The ARN form keeps
+the key out of Terraform variables, plan output and state: ``main.tf`` reads the parameter
+**without decryption** (only to prove it exists) and passes its ARN; the stack's service role is
+the only principal that decrypts it. Secrets Manager is not supported by the template.
+
+.. code-block:: bash
+
+ aws ssm put-parameter --region us-east-2 \
+ --name /runs-on/license-key \
+ --type SecureString \
+ --description "RunsOn license key for the runs-on stack (GitHub organization apache)" \
+ --value 'RUNS-ON-LICENSE-KEY'
+
+Use the default SSM KMS key (``alias/aws/ssm``): the template grants the stack's role access to
+that key; a customer key would need an extra key policy. Never put the key in ``terraform.tfvars``.
+Rotating the key is ``aws ssm put-parameter --overwrite`` followed by a stack update.
+
+The free open-source license is issued per GitHub organization; the key must be the one issued
+for ``apache``.
+
+Step 2 — plan and apply
+-----------------------
+
+For the first, local run, keep the shared S3 state out of the way with a gitignored override::
+
+ cd terraform/runs-on
+ printf 'terraform {\n backend "local" {}\n}\n' > backend_override.tf
+ terraform init
+ terraform plan -out=runs-on.tfplan
+ terraform apply runs-on.tfplan # ~10 minutes
+ terraform output
+
+The plan must show exactly one resource to create (``aws_cloudformation_stack.runs_on``) and read
+one data source (the SSM parameter). Delete ``backend_override.tf`` and run ``terraform init
+-migrate-state`` to move the state to S3 once the future GitHub Actions workflow takes over;
+the workflow only needs an OIDC role, nothing secret.
+
+Step 3 — register the GitHub App and the presets
+------------------------------------------------
+
+1. Open the ``entry_point`` output → **Register app**. This creates a private GitHub App in the
+ organization and installs it; for ``apache`` this is done together with ASF Infra, who also
+ grant the app access to ``apache/airflow`` and to ``.github-private`` if the presets live there.
+2. Commit ``runs-on.yml.example`` as ``.github/runs-on.yml`` in ``apache/.github-private``
+ (stack parameter ``RunnerConfigAutoExtendsFrom``, default ``.github-private``). RunsOn reads
+ ``.github/runs-on.yml`` in a public repository **only from its default branch**, so presets
+ cannot be trialled from a feature branch; the validation therefore spelled the same values
+ out in the labels.
+3. Point jobs at the runners:
+
+ .. code-block:: yaml
+
+ runs-on: runs-on=${{ github.run_id }}/runner=airflow-amd-medium
+
+ ``runs-on=<run id>`` is mandatory (it routes the job and namespaces the runner). Labels can
+ also carry everything inline, which is what the validation did:
+ ``runs-on=<id>/family=m7a+m7i+m6a+m6i/image=ubuntu22-full-x64/volume=150gb/cpu=8/ram=32``.
+ ``runs-on/action`` (per-job cost line in the job summary) is already on the ASF GitHub Actions
+ allowlist.
+
+Upgrades and changes
+--------------------
+
+Bump ``runs_on_version`` (or any parameter) and ``terraform plan`` / ``apply``: the resource
+always resubmits the template URL, which is what RunsOn's `upgrade guide
+<https://runs-on.com/docs/maintenance/upgrades/>`_ requires (a reused processed template can
+silently skip parameter changes). Minor and patch releases have no downtime; major releases may.
+Rollback is the previous ``runs_on_version``. Template ``v3.3.2`` has SHA-256
+``3017805d77c1d717fdd229a1e1242db12c985944699d3d8dca01f6f28b35c16d``.
+
+``prevent_destroy`` is set on the stack: a ``terraform destroy`` fails until the lifecycle block is
+removed on purpose. Deleting the stack also needs its cache bucket emptied first if CloudFormation
+stops on it.
+
+What the stack contains
+-----------------------
+
+VPC ``10.2.0.0/16`` with public and private subnets (``Private=false`` → runners get public IPs and
+no NAT, so each runner pulls from Docker Hub from its own address), an ECS Fargate service for the
+control plane, an API Gateway entry point (``/``, ``/setup``, ``/readyz`` only), an S3 cache bucket
+(``cache_bucket`` output; the runners' instance role can read and write the ``cache/`` prefix,
+lifecycle ``S3CacheExpirationInDays``), the EC2 instance role and launch templates, DynamoDB
+tables for locks and job tracking, an SNS alert topic, an AWS Budget (``AppBudgetDailyUsd``,
+alert only — not a spending cap), the CloudWatch dashboard ``RunsOnDashboard-*`` (queue health,
+scheduling trend, queue-duration P50/P90, Spot circuit breaker), log groups
+``/aws/ecs/<stack>/runs-on-worker`` and ``<stack>-EC2InstanceLogGroup-*``, and a daily cost report
+by email.
+
+Observability
+-------------
+
+- CloudWatch dashboard and log groups above; ``roc`` CLI (`docs
+ <https://runs-on.com/docs/observability/cli/>`_): ``roc stack doctor``, ``roc stack logs``,
+ ``roc logs <job url> --watch``, ``roc connect <job url>`` over SSM (``SSMAllowed=true``).
+- Optional OTLP export (``OtelExporterEndpoint`` / ``OtelExporterHeaders`` + ``extras=otel`` on the
+ label) to SigNoz or any OTLP backend; RunsOn publishes a SigNoz dashboard. Not deployed by
+ the stack.
+
+Things learned in the validation
+--------------------------------
+
+- Runner instances use IMDSv2 with a hop limit of 2, so Docker containers started by a job can
+ obtain the instance role's credentials. Tests that assume "no AWS credentials" must set
Review Comment:
Stated in the README now ("Things learned in the validation"):
- Untrusted code is meant to stay off these runners. apache/airflow#74889 routes only pushes, schedules and dispatches on `apache/airflow`, branches of `apache/airflow` itself, and pull requests labelled `use self-hosted runners` that a committer approved at their head commit. Everything else stays on GitHub-hosted runners. The README also says this routing is cooperative, not a boundary: a fork's `pull_request` run uses the fork's workflow file. The actual boundary is GitHub's fork-workflow approval set to "Require approval for all external contributors".
- Stash payloads can't be poisoned through the bucket. They are stored by sha256 (`cache/airflow-stash/sha256/<digest>.tar`), the digest itself is kept as a GitHub-scoped ASF stash, and the restore checks the download against it. An overwritten object becomes a cache miss, not a different restore.
I'm leaving this thread open on purpose. The boundary depends on two things that aren't in place yet: the "Require approval for all external contributors" setting on `apache/airflow` (ASF Infra) and the merge of apache/airflow#74889. Until both are done, nothing in this repository can enforce it.
---
Drafted-by: Claude Code (Opus 5.5) (no human review before posting)
##########
terraform/runs-on/README.rst:
##########
@@ -0,0 +1,165 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+RunsOn — self-hosted CI runners on AWS
+======================================
+
+`RunsOn <https://runs-on.com/>`_ is the approach selected by
+`AIP-118 <https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974652>`_ for Airflow's
+self-hosted GitHub Actions runners. It packages runner provisioning into a CloudFormation stack in
+Airflow's AWS account: a small control plane (ECS Fargate + API Gateway) receives GitHub's
+``workflow_job`` webhooks and launches one ephemeral EC2 instance per job, Spot first, sized from
+the job's ``runs-on:`` label. RunsOn's `free open-source license <https://runs-on.com/billing/free/>`_
+covers the product; EC2/EBS/S3 usage is billed to the account.
+
+This module creates the stack for the **apache** organization from RunsOn's published,
+version-pinned template, with the configuration validated on 2026-09-25 (full ``Tests (AMD)``
+graph green apart from two non-hermetic tests — see the AIP-118 appendix).
+
+Files
+-----
+
+``main.tf``
+ The ``aws_cloudformation_stack`` resource, its parameters and the license-key lookup.
+``variables.tf``
+ All variables have defaults: ``stack_name`` (``runs-on``), ``github_organization``
+ (``apache``), ``alert_email`` (``shahar@apache.org``), ``runs_on_version`` (``v3.3.2``),
+ ``vpc_cidr`` (``10.2.0.0/16``) and ``region`` (``us-east-2``).
+``state.tf``
+ S3 backend ``airflow-ci-tfstate`` / ``tf-state/runs-on.tfstate`` (its own key next to the
+ EKS states).
+``runs-on.yml.example``
+ Runner presets for Airflow's three size classes, to be committed as ``.github/runs-on.yml``
+ in the organization's ``.github-private`` repository (or a repository's default branch).
+
+Step 1 — create the license-key parameter (manual, once)
+--------------------------------------------------------
+
+RunsOn accepts ``LicenseKey`` either as plain text or as the ARN of an SSM Parameter Store
+parameter that must be named ``/<stack name>/license-key`` (`reference
+<https://runs-on.com/docs/maintenance/configuration/flex/cloudformation/>`_). The ARN form keeps
+the key out of Terraform variables, plan output and state: ``main.tf`` reads the parameter
+**without decryption** (only to prove it exists) and passes its ARN; the stack's service role is
+the only principal that decrypts it. Secrets Manager is not supported by the template.
+
+.. code-block:: bash
+
+ aws ssm put-parameter --region us-east-2 \
+ --name /runs-on/license-key \
+ --type SecureString \
+ --description "RunsOn license key for the runs-on stack (GitHub organization apache)" \
+ --value 'RUNS-ON-LICENSE-KEY'
+
+Use the default SSM KMS key (``alias/aws/ssm``): the template grants the stack's role access to
+that key; a customer key would need an extra key policy. Never put the key in ``terraform.tfvars``.
+Rotating the key is ``aws ssm put-parameter --overwrite`` followed by a stack update.
+
+The free open-source license is issued per GitHub organization; the key must be the one issued
+for ``apache``.
+
+Step 2 — plan and apply
+-----------------------
+
+For the first, local run, keep the shared S3 state out of the way with a gitignored override::
+
+ cd terraform/runs-on
+ printf 'terraform {\n backend "local" {}\n}\n' > backend_override.tf
+ terraform init
+ terraform plan -out=runs-on.tfplan
+ terraform apply runs-on.tfplan # ~10 minutes
+ terraform output
+
+The plan must show exactly one resource to create (``aws_cloudformation_stack.runs_on``) and read
+one data source (the SSM parameter). Delete ``backend_override.tf`` and run ``terraform init
+-migrate-state`` to move the state to S3 once the future GitHub Actions workflow takes over;
Review Comment:
Yes, it was applied with the local backend on 2026-09-25. The state is now migrated to S3 (`terraform init -migrate-state` into `tf-state/runs-on.tfstate`; the bucket is versioned and encrypted), so the committed backend is the real one. A plan from a clean checkout matches the stack and does not propose to create it.
The README (Step 2) now describes the S3 state and warns that a plan proposing to create the stack means Terraform isn't reading the S3 state.
---
Drafted-by: Claude Code (Opus 5.5) (no human review before posting)
##########
terraform/runs-on/README.rst:
##########
@@ -0,0 +1,165 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+RunsOn — self-hosted CI runners on AWS
+======================================
+
+`RunsOn <https://runs-on.com/>`_ is the approach selected by
+`AIP-118 <https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974652>`_ for Airflow's
+self-hosted GitHub Actions runners. It packages runner provisioning into a CloudFormation stack in
+Airflow's AWS account: a small control plane (ECS Fargate + API Gateway) receives GitHub's
+``workflow_job`` webhooks and launches one ephemeral EC2 instance per job, Spot first, sized from
+the job's ``runs-on:`` label. RunsOn's `free open-source license <https://runs-on.com/billing/free/>`_
+covers the product; EC2/EBS/S3 usage is billed to the account.
+
+This module creates the stack for the **apache** organization from RunsOn's published,
+version-pinned template, with the configuration validated on 2026-09-25 (full ``Tests (AMD)``
+graph green apart from two non-hermetic tests — see the AIP-118 appendix).
+
+Files
+-----
+
+``main.tf``
+ The ``aws_cloudformation_stack`` resource, its parameters and the license-key lookup.
+``variables.tf``
+ All variables have defaults: ``stack_name`` (``runs-on``), ``github_organization``
+ (``apache``), ``alert_email`` (``shahar@apache.org``), ``runs_on_version`` (``v3.3.2``),
+ ``vpc_cidr`` (``10.2.0.0/16``) and ``region`` (``us-east-2``).
+``state.tf``
+ S3 backend ``airflow-ci-tfstate`` / ``tf-state/runs-on.tfstate`` (its own key next to the
+ EKS states).
+``runs-on.yml.example``
+ Runner presets for Airflow's three size classes, to be committed as ``.github/runs-on.yml``
+ in the organization's ``.github-private`` repository (or a repository's default branch).
+
+Step 1 — create the license-key parameter (manual, once)
+--------------------------------------------------------
+
+RunsOn accepts ``LicenseKey`` either as plain text or as the ARN of an SSM Parameter Store
+parameter that must be named ``/<stack name>/license-key`` (`reference
+<https://runs-on.com/docs/maintenance/configuration/flex/cloudformation/>`_). The ARN form keeps
+the key out of Terraform variables, plan output and state: ``main.tf`` reads the parameter
+**without decryption** (only to prove it exists) and passes its ARN; the stack's service role is
+the only principal that decrypts it. Secrets Manager is not supported by the template.
+
+.. code-block:: bash
+
+ aws ssm put-parameter --region us-east-2 \
+ --name /runs-on/license-key \
+ --type SecureString \
+ --description "RunsOn license key for the runs-on stack (GitHub organization apache)" \
+ --value 'RUNS-ON-LICENSE-KEY'
+
+Use the default SSM KMS key (``alias/aws/ssm``): the template grants the stack's role access to
+that key; a customer key would need an extra key policy. Never put the key in ``terraform.tfvars``.
+Rotating the key is ``aws ssm put-parameter --overwrite`` followed by a stack update.
+
Review Comment:
Right, Terraform can't do it, so the README now documents the manual step. Overwrite the parameter, then restart the control-plane ECS service (`aws ecs update-service --force-new-deployment`; the exact commands are in Step 1). The template shows the service reads `/runs-on/license-key` itself at runtime, and that is the only principal allowed to.
One correction to the premise: the plan isn't empty. Every plan shows `"LicenseKey" = "****" -> "arn:..."`, because CloudFormation masks `NoEcho` parameters and the AWS provider stores the mask. The value sent is the same ARN, so that line alone changes nothing. The README explains this and warns against `ignore_changes`, which would make the provider send the literal `****`.
---
Drafted-by: Claude Code (Opus 5.5) (no human review before posting)
##########
terraform/runs-on/version.tf:
##########
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+terraform {
+ required_providers {
+ aws = {
+ source = "hashicorp/aws"
+ version = "~> 6.0"
+ }
+ }
+ # 1.10+ for S3-native state locking (use_lockfile)
+ required_version = ">= 1.10.0"
Review Comment:
Done in 87e1da6: `required_version = ">= 1.11.0"`. The upcoming workflow (#66) pins Terraform 1.16.2.
---
Drafted-by: Claude Code (Opus 5.5) (no human review before posting)
##########
terraform/runs-on/variables.tf:
##########
@@ -0,0 +1,55 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+variable "region" {
+ description = "AWS region for the RunsOn stack and its runners."
+ type = string
+ default = "us-east-2"
+}
+
+variable "stack_name" {
+ description = <<-EOT
+ CloudFormation stack name. RunsOn derives the license-key SSM parameter name from it
+ (/<stack_name>/license-key), so changing it means re-creating the parameter.
+ EOT
+ type = string
+ default = "runs-on"
+}
+
+variable "runs_on_version" {
+ description = "RunsOn release to deploy; selects the version-pinned CloudFormation template."
+ type = string
+ default = "v3.3.2"
+}
+
+variable "github_organization" {
+ description = "GitHub organization the runners serve (case-sensitive)."
+ type = string
+ default = "apache"
+}
+
+variable "alert_email" {
+ description = "Recipient of RunsOn cost reports and alerts; AWS sends an SNS confirmation email on creation."
+ type = string
+ default = "shahar@apache.org"
Review Comment:
Done in 87e1da6 and applied: `alert_email` defaults to `private@airflow.apache.org`, and the stack's `EmailAddress` was updated.
Not finished yet, though. private@ only accepts mail from subscribers, so the SNS confirmation from `no-reply@sns.amazonaws.com` was held for moderation and never reached the list. The private@ subscription is still `PendingConfirmation`. Every alert and daily cost report comes from the same sender, so the fix is to allow-list `no-reply@sns.amazonaws.com` on private@ (a list moderator's task), re-send the confirmation with `aws sns subscribe`, and confirm it. Until then, alerts still go to the old personal subscription, so nothing is lost. That subscription will be removed once the list one is confirmed.
---
Drafted-by: Claude Code (Opus 5.5) (no human review before posting)
##########
terraform/runs-on/main.tf:
##########
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
Review Comment:
Removed both in 87e1da6. The stack module now reads only the SSM parameter, which matches the README.
#66 brings back `aws_caller_identity` on purpose: it builds the ARN of the new CloudFormation service role. The README there says the plan reads two data sources.
---
Drafted-by: Claude Code (Opus 5.5) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Terraform for the RunsOn stack of the apache organization [airflow-ci-infra]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #65:
URL: https://github.com/apache/airflow-ci-infra/pull/65#issuecomment-5833723165
looks really simple!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Terraform for the RunsOn stack of the apache organization [airflow-ci-infra]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #65:
URL: https://github.com/apache/airflow-ci-infra/pull/65#discussion_r4353978165
##########
terraform/runs-on/main.tf:
##########
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+data "aws_partition" "current" {}
+data "aws_caller_identity" "current" {}
+
+# The license key is never in Terraform state: RunsOn accepts LicenseKey as the ARN of an SSM
+# SecureString parameter that MUST be named /<stack name>/license-key, and only the stack's
+# service role decrypts it. The parameter is created by hand (see README.rst); this lookup
+# without decryption only proves it exists before the stack is planned, storing the ciphertext.
+data "aws_ssm_parameter" "license_key" {
+ name = "/${var.stack_name}/license-key"
+ with_decryption = false
+}
+
+resource "aws_cloudformation_stack" "runs_on" {
+ name = var.stack_name
+ template_url = "https://runs-on.s3.eu-west-1.amazonaws.com/cloudformation/template-${var.runs_on_version}.yaml"
+ capabilities = ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM", "CAPABILITY_AUTO_EXPAND"]
+
+ # Everything not listed keeps the template default; the values match the stack validated on
+ # 2026-09-25 (see the AIP-118 appendix) apart from the organization and the CIDR.
+ parameters = {
+ GithubOrganization = var.github_organization
+ LicenseKey = data.aws_ssm_parameter.license_key.arn
+ EmailAddress = var.alert_email
+ Environment = "production"
+ VpcCidrBlock = var.vpc_cidr
+ Private = "false" # runners get public IPs: no NAT cost, Docker Hub pulls per runner IP
+ SSHAllowed = "false"
+ SSMAllowed = "true"
Review Comment:
This keeps `/setup` reachable on the public API Gateway for the life of the stack, and every apply turns it back on. Could we make it a variable defaulting to `false`, set to `true` only for the GitHub App registration (and for re-registering if that's ever needed)?
##########
terraform/runs-on/README.rst:
##########
@@ -0,0 +1,165 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+RunsOn — self-hosted CI runners on AWS
+======================================
+
+`RunsOn <https://runs-on.com/>`_ is the approach selected by
+`AIP-118 <https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974652>`_ for Airflow's
+self-hosted GitHub Actions runners. It packages runner provisioning into a CloudFormation stack in
+Airflow's AWS account: a small control plane (ECS Fargate + API Gateway) receives GitHub's
+``workflow_job`` webhooks and launches one ephemeral EC2 instance per job, Spot first, sized from
+the job's ``runs-on:`` label. RunsOn's `free open-source license <https://runs-on.com/billing/free/>`_
+covers the product; EC2/EBS/S3 usage is billed to the account.
+
+This module creates the stack for the **apache** organization from RunsOn's published,
+version-pinned template, with the configuration validated on 2026-09-25 (full ``Tests (AMD)``
+graph green apart from two non-hermetic tests — see the AIP-118 appendix).
+
+Files
+-----
+
+``main.tf``
+ The ``aws_cloudformation_stack`` resource, its parameters and the license-key lookup.
+``variables.tf``
+ All variables have defaults: ``stack_name`` (``runs-on``), ``github_organization``
+ (``apache``), ``alert_email`` (``shahar@apache.org``), ``runs_on_version`` (``v3.3.2``),
+ ``vpc_cidr`` (``10.2.0.0/16``) and ``region`` (``us-east-2``).
+``state.tf``
+ S3 backend ``airflow-ci-tfstate`` / ``tf-state/runs-on.tfstate`` (its own key next to the
+ EKS states).
+``runs-on.yml.example``
+ Runner presets for Airflow's three size classes, to be committed as ``.github/runs-on.yml``
+ in the organization's ``.github-private`` repository (or a repository's default branch).
+
+Step 1 — create the license-key parameter (manual, once)
+--------------------------------------------------------
+
+RunsOn accepts ``LicenseKey`` either as plain text or as the ARN of an SSM Parameter Store
+parameter that must be named ``/<stack name>/license-key`` (`reference
+<https://runs-on.com/docs/maintenance/configuration/flex/cloudformation/>`_). The ARN form keeps
+the key out of Terraform variables, plan output and state: ``main.tf`` reads the parameter
+**without decryption** (only to prove it exists) and passes its ARN; the stack's service role is
+the only principal that decrypts it. Secrets Manager is not supported by the template.
+
+.. code-block:: bash
+
+ aws ssm put-parameter --region us-east-2 \
+ --name /runs-on/license-key \
+ --type SecureString \
+ --description "RunsOn license key for the runs-on stack (GitHub organization apache)" \
+ --value 'RUNS-ON-LICENSE-KEY'
+
+Use the default SSM KMS key (``alias/aws/ssm``): the template grants the stack's role access to
+that key; a customer key would need an extra key policy. Never put the key in ``terraform.tfvars``.
+Rotating the key is ``aws ssm put-parameter --overwrite`` followed by a stack update.
+
+The free open-source license is issued per GitHub organization; the key must be the one issued
+for ``apache``.
+
+Step 2 — plan and apply
+-----------------------
+
+For the first, local run, keep the shared S3 state out of the way with a gitignored override::
+
+ cd terraform/runs-on
+ printf 'terraform {\n backend "local" {}\n}\n' > backend_override.tf
+ terraform init
+ terraform plan -out=runs-on.tfplan
+ terraform apply runs-on.tfplan # ~10 minutes
+ terraform output
+
+The plan must show exactly one resource to create (``aws_cloudformation_stack.runs_on``) and read
+one data source (the SSM parameter). Delete ``backend_override.tf`` and run ``terraform init
+-migrate-state`` to move the state to S3 once the future GitHub Actions workflow takes over;
Review Comment:
Is the `apache` stack already applied with the local backend? If so, the committed S3 key is empty until someone runs `-migrate-state`. Anyone who runs `terraform init`/`plan` without the override in the meantime gets a plan to create the stack, and apply then fails with `AlreadyExistsException`. Meanwhile the only copy of the state is on one machine. Could we migrate it to S3 before merging, or at least say clearly here that the migration is pending?
##########
terraform/runs-on/README.rst:
##########
@@ -0,0 +1,165 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+RunsOn — self-hosted CI runners on AWS
+======================================
+
+`RunsOn <https://runs-on.com/>`_ is the approach selected by
+`AIP-118 <https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974652>`_ for Airflow's
+self-hosted GitHub Actions runners. It packages runner provisioning into a CloudFormation stack in
+Airflow's AWS account: a small control plane (ECS Fargate + API Gateway) receives GitHub's
+``workflow_job`` webhooks and launches one ephemeral EC2 instance per job, Spot first, sized from
+the job's ``runs-on:`` label. RunsOn's `free open-source license <https://runs-on.com/billing/free/>`_
+covers the product; EC2/EBS/S3 usage is billed to the account.
+
+This module creates the stack for the **apache** organization from RunsOn's published,
+version-pinned template, with the configuration validated on 2026-09-25 (full ``Tests (AMD)``
+graph green apart from two non-hermetic tests — see the AIP-118 appendix).
+
+Files
+-----
+
+``main.tf``
+ The ``aws_cloudformation_stack`` resource, its parameters and the license-key lookup.
+``variables.tf``
+ All variables have defaults: ``stack_name`` (``runs-on``), ``github_organization``
+ (``apache``), ``alert_email`` (``shahar@apache.org``), ``runs_on_version`` (``v3.3.2``),
+ ``vpc_cidr`` (``10.2.0.0/16``) and ``region`` (``us-east-2``).
+``state.tf``
+ S3 backend ``airflow-ci-tfstate`` / ``tf-state/runs-on.tfstate`` (its own key next to the
+ EKS states).
+``runs-on.yml.example``
+ Runner presets for Airflow's three size classes, to be committed as ``.github/runs-on.yml``
+ in the organization's ``.github-private`` repository (or a repository's default branch).
+
+Step 1 — create the license-key parameter (manual, once)
+--------------------------------------------------------
+
+RunsOn accepts ``LicenseKey`` either as plain text or as the ARN of an SSM Parameter Store
+parameter that must be named ``/<stack name>/license-key`` (`reference
+<https://runs-on.com/docs/maintenance/configuration/flex/cloudformation/>`_). The ARN form keeps
+the key out of Terraform variables, plan output and state: ``main.tf`` reads the parameter
+**without decryption** (only to prove it exists) and passes its ARN; the stack's service role is
+the only principal that decrypts it. Secrets Manager is not supported by the template.
+
+.. code-block:: bash
+
+ aws ssm put-parameter --region us-east-2 \
+ --name /runs-on/license-key \
+ --type SecureString \
+ --description "RunsOn license key for the runs-on stack (GitHub organization apache)" \
+ --value 'RUNS-ON-LICENSE-KEY'
+
+Use the default SSM KMS key (``alias/aws/ssm``): the template grants the stack's role access to
+that key; a customer key would need an extra key policy. Never put the key in ``terraform.tfvars``.
+Rotating the key is ``aws ssm put-parameter --overwrite`` followed by a stack update.
+
Review Comment:
After `put-parameter --overwrite`, `LicenseKey` is still the same ARN, so `terraform plan` shows no changes and Terraform never triggers a stack update. How is the "stack update" meant to happen? A restart of the ECS service, or a documented manual step?
##########
terraform/runs-on/outputs.tf:
##########
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+output "entry_point" {
+ description = "RunsOn web entry point: open it once after creation to register the GitHub App."
+ value = aws_cloudformation_stack.runs_on.outputs["RunsOnEntryPoint"]
+}
+
+output "cache_bucket" {
+ description = "S3 bucket the runners can use under the cache/ prefix (stash payloads)."
+ value = aws_cloudformation_stack.runs_on.outputs["RunsOnBucketCache"]
+}
+
+output "app_version" {
Review Comment:
Nit: these two outputs have no `description`, unlike the other two.
##########
terraform/runs-on/variables.tf:
##########
@@ -0,0 +1,55 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+variable "region" {
+ description = "AWS region for the RunsOn stack and its runners."
+ type = string
+ default = "us-east-2"
+}
+
+variable "stack_name" {
+ description = <<-EOT
+ CloudFormation stack name. RunsOn derives the license-key SSM parameter name from it
+ (/<stack_name>/license-key), so changing it means re-creating the parameter.
+ EOT
+ type = string
+ default = "runs-on"
+}
+
+variable "runs_on_version" {
+ description = "RunsOn release to deploy; selects the version-pinned CloudFormation template."
+ type = string
+ default = "v3.3.2"
+}
+
+variable "github_organization" {
+ description = "GitHub organization the runners serve (case-sensitive)."
+ type = string
+ default = "apache"
+}
+
+variable "alert_email" {
+ description = "Recipient of RunsOn cost reports and alerts; AWS sends an SNS confirmation email on creation."
+ type = string
+ default = "shahar@apache.org"
Review Comment:
Defaulting alerts, budget notices and cost reports to a personal address ties them to one person's inbox and SNS confirmation. Could this be a list address (for example a private@ list, or an ci-infra alias)?
##########
terraform/runs-on/README.rst:
##########
@@ -0,0 +1,165 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+RunsOn — self-hosted CI runners on AWS
+======================================
+
+`RunsOn <https://runs-on.com/>`_ is the approach selected by
+`AIP-118 <https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974652>`_ for Airflow's
+self-hosted GitHub Actions runners. It packages runner provisioning into a CloudFormation stack in
+Airflow's AWS account: a small control plane (ECS Fargate + API Gateway) receives GitHub's
+``workflow_job`` webhooks and launches one ephemeral EC2 instance per job, Spot first, sized from
+the job's ``runs-on:`` label. RunsOn's `free open-source license <https://runs-on.com/billing/free/>`_
+covers the product; EC2/EBS/S3 usage is billed to the account.
+
+This module creates the stack for the **apache** organization from RunsOn's published,
+version-pinned template, with the configuration validated on 2026-09-25 (full ``Tests (AMD)``
+graph green apart from two non-hermetic tests — see the AIP-118 appendix).
+
+Files
+-----
+
+``main.tf``
+ The ``aws_cloudformation_stack`` resource, its parameters and the license-key lookup.
+``variables.tf``
+ All variables have defaults: ``stack_name`` (``runs-on``), ``github_organization``
+ (``apache``), ``alert_email`` (``shahar@apache.org``), ``runs_on_version`` (``v3.3.2``),
+ ``vpc_cidr`` (``10.2.0.0/16``) and ``region`` (``us-east-2``).
+``state.tf``
+ S3 backend ``airflow-ci-tfstate`` / ``tf-state/runs-on.tfstate`` (its own key next to the
+ EKS states).
+``runs-on.yml.example``
+ Runner presets for Airflow's three size classes, to be committed as ``.github/runs-on.yml``
+ in the organization's ``.github-private`` repository (or a repository's default branch).
+
+Step 1 — create the license-key parameter (manual, once)
+--------------------------------------------------------
+
+RunsOn accepts ``LicenseKey`` either as plain text or as the ARN of an SSM Parameter Store
+parameter that must be named ``/<stack name>/license-key`` (`reference
+<https://runs-on.com/docs/maintenance/configuration/flex/cloudformation/>`_). The ARN form keeps
+the key out of Terraform variables, plan output and state: ``main.tf`` reads the parameter
+**without decryption** (only to prove it exists) and passes its ARN; the stack's service role is
+the only principal that decrypts it. Secrets Manager is not supported by the template.
+
+.. code-block:: bash
+
+ aws ssm put-parameter --region us-east-2 \
+ --name /runs-on/license-key \
+ --type SecureString \
+ --description "RunsOn license key for the runs-on stack (GitHub organization apache)" \
+ --value 'RUNS-ON-LICENSE-KEY'
+
+Use the default SSM KMS key (``alias/aws/ssm``): the template grants the stack's role access to
+that key; a customer key would need an extra key policy. Never put the key in ``terraform.tfvars``.
+Rotating the key is ``aws ssm put-parameter --overwrite`` followed by a stack update.
+
+The free open-source license is issued per GitHub organization; the key must be the one issued
+for ``apache``.
+
+Step 2 — plan and apply
+-----------------------
+
+For the first, local run, keep the shared S3 state out of the way with a gitignored override::
+
+ cd terraform/runs-on
+ printf 'terraform {\n backend "local" {}\n}\n' > backend_override.tf
+ terraform init
+ terraform plan -out=runs-on.tfplan
+ terraform apply runs-on.tfplan # ~10 minutes
+ terraform output
+
+The plan must show exactly one resource to create (``aws_cloudformation_stack.runs_on``) and read
+one data source (the SSM parameter). Delete ``backend_override.tf`` and run ``terraform init
+-migrate-state`` to move the state to S3 once the future GitHub Actions workflow takes over;
+the workflow only needs an OIDC role, nothing secret.
+
+Step 3 — register the GitHub App and the presets
+------------------------------------------------
+
+1. Open the ``entry_point`` output → **Register app**. This creates a private GitHub App in the
+ organization and installs it; for ``apache`` this is done together with ASF Infra, who also
+ grant the app access to ``apache/airflow`` and to ``.github-private`` if the presets live there.
+2. Commit ``runs-on.yml.example`` as ``.github/runs-on.yml`` in ``apache/.github-private``
+ (stack parameter ``RunnerConfigAutoExtendsFrom``, default ``.github-private``). RunsOn reads
+ ``.github/runs-on.yml`` in a public repository **only from its default branch**, so presets
+ cannot be trialled from a feature branch; the validation therefore spelled the same values
+ out in the labels.
+3. Point jobs at the runners:
+
+ .. code-block:: yaml
+
+ runs-on: runs-on=${{ github.run_id }}/runner=airflow-amd-medium
+
+ ``runs-on=<run id>`` is mandatory (it routes the job and namespaces the runner). Labels can
+ also carry everything inline, which is what the validation did:
+ ``runs-on=<id>/family=m7a+m7i+m6a+m6i/image=ubuntu22-full-x64/volume=150gb/cpu=8/ram=32``.
+ ``runs-on/action`` (per-job cost line in the job summary) is already on the ASF GitHub Actions
+ allowlist.
+
+Upgrades and changes
+--------------------
+
+Bump ``runs_on_version`` (or any parameter) and ``terraform plan`` / ``apply``: the resource
+always resubmits the template URL, which is what RunsOn's `upgrade guide
+<https://runs-on.com/docs/maintenance/upgrades/>`_ requires (a reused processed template can
+silently skip parameter changes). Minor and patch releases have no downtime; major releases may.
+Rollback is the previous ``runs_on_version``. Template ``v3.3.2`` has SHA-256
+``3017805d77c1d717fdd229a1e1242db12c985944699d3d8dca01f6f28b35c16d``.
+
+``prevent_destroy`` is set on the stack: a ``terraform destroy`` fails until the lifecycle block is
+removed on purpose. Deleting the stack also needs its cache bucket emptied first if CloudFormation
+stops on it.
+
+What the stack contains
+-----------------------
+
+VPC ``10.2.0.0/16`` with public and private subnets (``Private=false`` → runners get public IPs and
+no NAT, so each runner pulls from Docker Hub from its own address), an ECS Fargate service for the
+control plane, an API Gateway entry point (``/``, ``/setup``, ``/readyz`` only), an S3 cache bucket
+(``cache_bucket`` output; the runners' instance role can read and write the ``cache/`` prefix,
+lifecycle ``S3CacheExpirationInDays``), the EC2 instance role and launch templates, DynamoDB
+tables for locks and job tracking, an SNS alert topic, an AWS Budget (``AppBudgetDailyUsd``,
+alert only — not a spending cap), the CloudWatch dashboard ``RunsOnDashboard-*`` (queue health,
+scheduling trend, queue-duration P50/P90, Spot circuit breaker), log groups
+``/aws/ecs/<stack>/runs-on-worker`` and ``<stack>-EC2InstanceLogGroup-*``, and a daily cost report
+by email.
+
+Observability
+-------------
+
+- CloudWatch dashboard and log groups above; ``roc`` CLI (`docs
+ <https://runs-on.com/docs/observability/cli/>`_): ``roc stack doctor``, ``roc stack logs``,
+ ``roc logs <job url> --watch``, ``roc connect <job url>`` over SSM (``SSMAllowed=true``).
+- Optional OTLP export (``OtelExporterEndpoint`` / ``OtelExporterHeaders`` + ``extras=otel`` on the
+ label) to SigNoz or any OTLP backend; RunsOn publishes a SigNoz dashboard. Not deployed by
+ the stack.
+
+Things learned in the validation
+--------------------------------
+
+- Runner instances use IMDSv2 with a hop limit of 2, so Docker containers started by a job can
+ obtain the instance role's credentials. Tests that assume "no AWS credentials" must set
Review Comment:
Beyond test hermeticity, this means any job container can get the instance role's credentials, and that role can write `cache/`, including `cache/airflow-stash/`. If PRs from forks ever run on these runners, a PR could overwrite stash payloads that a later main-branch run restores. Do we plan to keep untrusted PRs off these runners, or to separate cache prefixes by trust level? Worth stating here either way.
##########
terraform/runs-on/version.tf:
##########
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+terraform {
+ required_providers {
+ aws = {
+ source = "hashicorp/aws"
+ version = "~> 6.0"
+ }
+ }
+ # 1.10+ for S3-native state locking (use_lockfile)
+ required_version = ">= 1.10.0"
Review Comment:
S3 native locking (`use_lockfile`) was experimental in 1.10 and became GA in 1.11. Since this state will be shared with a CI workflow, `>= 1.11.0` seems safer.
##########
terraform/runs-on/README.rst:
##########
@@ -0,0 +1,165 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+RunsOn — self-hosted CI runners on AWS
+======================================
+
+`RunsOn <https://runs-on.com/>`_ is the approach selected by
+`AIP-118 <https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974652>`_ for Airflow's
+self-hosted GitHub Actions runners. It packages runner provisioning into a CloudFormation stack in
+Airflow's AWS account: a small control plane (ECS Fargate + API Gateway) receives GitHub's
+``workflow_job`` webhooks and launches one ephemeral EC2 instance per job, Spot first, sized from
+the job's ``runs-on:`` label. RunsOn's `free open-source license <https://runs-on.com/billing/free/>`_
+covers the product; EC2/EBS/S3 usage is billed to the account.
+
+This module creates the stack for the **apache** organization from RunsOn's published,
+version-pinned template, with the configuration validated on 2026-09-25 (full ``Tests (AMD)``
+graph green apart from two non-hermetic tests — see the AIP-118 appendix).
+
+Files
+-----
+
+``main.tf``
+ The ``aws_cloudformation_stack`` resource, its parameters and the license-key lookup.
+``variables.tf``
+ All variables have defaults: ``stack_name`` (``runs-on``), ``github_organization``
+ (``apache``), ``alert_email`` (``shahar@apache.org``), ``runs_on_version`` (``v3.3.2``),
+ ``vpc_cidr`` (``10.2.0.0/16``) and ``region`` (``us-east-2``).
+``state.tf``
+ S3 backend ``airflow-ci-tfstate`` / ``tf-state/runs-on.tfstate`` (its own key next to the
+ EKS states).
+``runs-on.yml.example``
+ Runner presets for Airflow's three size classes, to be committed as ``.github/runs-on.yml``
+ in the organization's ``.github-private`` repository (or a repository's default branch).
+
+Step 1 — create the license-key parameter (manual, once)
+--------------------------------------------------------
+
+RunsOn accepts ``LicenseKey`` either as plain text or as the ARN of an SSM Parameter Store
+parameter that must be named ``/<stack name>/license-key`` (`reference
+<https://runs-on.com/docs/maintenance/configuration/flex/cloudformation/>`_). The ARN form keeps
+the key out of Terraform variables, plan output and state: ``main.tf`` reads the parameter
+**without decryption** (only to prove it exists) and passes its ARN; the stack's service role is
+the only principal that decrypts it. Secrets Manager is not supported by the template.
+
+.. code-block:: bash
+
+ aws ssm put-parameter --region us-east-2 \
+ --name /runs-on/license-key \
Review Comment:
`var.region` looks configurable, but this command and the backend in `state.tf` both hard-code `us-east-2`. With any other region, the SSM lookup fails with `ParameterNotFound`. Either drop the variable or mention that the region has to be changed in these places too.
##########
terraform/runs-on/main.tf:
##########
@@ -0,0 +1,62 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
Review Comment:
These two data sources aren't used anywhere. The README (L87) also says the plan reads only one data source, the SSM parameter, so whoever follows that check will see three. I suggest removing them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Terraform for the RunsOn stack of the apache organization [airflow-ci-infra]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #65:
URL: https://github.com/apache/airflow-ci-infra/pull/65
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Terraform for the RunsOn stack of the apache organization [airflow-ci-infra]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #65:
URL: https://github.com/apache/airflow-ci-infra/pull/65#issuecomment-6365327251
All review comments are addressed in 87e1da6 (replies in the threads). The changes are applied to the `apache` stack, and the state now lives in S3.
The Terraform plan and apply workflow is the next PR in this stack: #66. In short:
- Plans run on pull requests that touch `terraform/runs-on` and again on `main` after merge. The `main` plan is applied only after a required reviewer approves the `runs-on-prod` environment (defined in `.asf.yaml`, main only, no self-approval).
- It authenticates with GitHub OIDC, so no AWS credential is stored anywhere. The roles are in `terraform/runs-on-access`, which an account admin applies by hand, so the workflow can't widen its own permissions:
- a read-only plan role (`kms:Decrypt` denied, so the license key stays unreadable);
- an apply role that can only resubmit the `runs-on` stack from RunsOn's published templates;
- a CloudFormation service role with explicit denies outside the stack's own resources.
- The trust policies use GitHub's immutable OIDC subject (`repo:apache@47359/airflow-ci-infra@321691041:...`), as `apache/reqsign` does. A renamed or re-created repository can't match them.
That last point needs a repository setting only ASF Infra can change: opting this repository into immutable OIDC subjects (`use_immutable_subject`). It is requested in [INFRA-28572](https://issues.apache.org/jira/browse/INFRA-28572). Until Infra approves and applies it, the workflow in #66 can't assume its AWS roles.
---
Drafted-by: Claude Code (Opus 5.5) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org