You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/11 11:15:25 UTC
[GitHub] [airflow] ashb commented on issue #14656: Save cost for CI infrastructure by having more types of build machines
ashb commented on issue #14656:
URL: https://github.com/apache/airflow/issues/14656#issuecomment-796661760
Rough steps we will need to do to have two "pools" of runners:
1. Get ASF infra to adjust the labels on the existing runners we have configured. Currently they just have `self-hosted` label -- we should ask them to set all 100 to additionally have a `large` label or something
1. Update https://github.com/apache/airflow-ci-infra/blob/master/scripts/runner-supervisor.py to pull credentials from a different path in AWS SSM.
Right now it looks for creds in `/runners/apache/airflow` -- i.e. `/runners/apache/airflow/1`...`/runners/apache/airflow/100`
We would probably have to change that to look at `/runners/apache/airflow/{size}/{idx}`
1. Open a ticket with Infra, and ask them to create a new runner registration token. Use that token and the https://github.com/apache/airflow-ci-infra/blob/master/scripts/store-agent-creds.py script to store those creds in SSM. Script will need updating to change/add labels and store in a different location too.
1. Create a second ASG with smaller instances, also configuring/changing the userdata so the instance starts in the right "pool".
1. The Lambda function will need updating to know which ASG to scale at based on the job name. I _think_ we might have to just duplicate the job ids and sizes in to the runner, as we don't get much info in the webhook payload (I've included a sample one below in the details block)
There may be a better approach here.
I'm more than happy to help or do all of the above myself if you'd like, just laying out the steps that come to mind.
<details>
<summary>Webhook payload</summary>
<pre><code>{
"action": "created",
"check_run": {
"id": 2085234698,
"node_id": "MDg6Q2hlY2tSdW4yMDg1MjM0Njk4",
"head_sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"external_id": "ff9e9115-6980-59bb-9ae7-308d64bdf685",
"url": "https://api.github.com/repos/apache/airflow/check-runs/2085234698",
"html_url": "https://github.com/apache/airflow/runs/2085234698",
"details_url": "https://github.com/apache/airflow/runs/2085234698",
"status": "queued",
"conclusion": null,
"started_at": "2021-03-11T10:09:49Z",
"completed_at": null,
"output": {
"title": null,
"summary": null,
"text": null,
"annotations_count": 0,
"annotations_url": "https://api.github.com/repos/apache/airflow/check-runs/2085234698/annotations"
},
"name": "K8s 3.7 v1.16.9 image",
"check_suite": {
"id": 2233450802,
"node_id": "MDEwOkNoZWNrU3VpdGUyMjMzNDUwODAy",
"head_branch": "master",
"head_sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"status": "queued",
"conclusion": null,
"url": "https://api.github.com/repos/apache/airflow/check-suites/2233450802",
"before": "943baff6701f9f8591090bf76219571d7f5e2cc5",
"after": "9070ba280253e918c2582f52c2b77a07c311198f",
"pull_requests": [
{
"url": "https://api.github.com/repos/rpatil524/airflow/pulls/80",
"id": 581521861,
"number": 80,
"head": {
"ref": "master",
"sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"repo": {
"id": 33884891,
"url": "https://api.github.com/repos/apache/airflow",
"name": "airflow"
}
},
"base": {
"ref": "master",
"sha": "fff344443f764a0a5c9f5ba7dfbfa1e5e1115348",
"repo": {
"id": 336576614,
"url": "https://api.github.com/repos/rpatil524/airflow",
"name": "airflow"
}
}
},
{
"url": "https://api.github.com/repos/yufeixin/airflow/pulls/305",
"id": 581371796,
"number": 305,
"head": {
"ref": "master",
"sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"repo": {
"id": 33884891,
"url": "https://api.github.com/repos/apache/airflow",
"name": "airflow"
}
},
"base": {
"ref": "master",
"sha": "f9cc775adc0dff49b0e288d8f2745bf097017321",
"repo": {
"id": 320438482,
"url": "https://api.github.com/repos/yufeixin/airflow",
"name": "airflow"
}
}
},
{
"url": "https://api.github.com/repos/ddelange/airflow/pulls/6",
"id": 571967064,
"number": 6,
"head": {
"ref": "master",
"sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"repo": {
"id": 33884891,
"url": "https://api.github.com/repos/apache/airflow",
"name": "airflow"
}
},
"base": {
"ref": "master",
"sha": "f9c9e9c38f444a39987478f3d1a262db909de8c4",
"repo": {
"id": 219798743,
"url": "https://api.github.com/repos/ddelange/airflow",
"name": "airflow"
}
}
},
{
"url": "https://api.github.com/repos/weiplanet/airflow/pulls/1",
"id": 562187084,
"number": 1,
"head": {
"ref": "master",
"sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"repo": {
"id": 33884891,
"url": "https://api.github.com/repos/apache/airflow",
"name": "airflow"
}
},
"base": {
"ref": "master",
"sha": "2345cd1f0337f4f06a2f2595a1a59c30dc4a1535",
"repo": {
"id": 303579760,
"url": "https://api.github.com/repos/weiplanet/airflow",
"name": "airflow"
}
}
},
{
"url": "https://api.github.com/repos/TestingCodeReview/airflow/pulls/1",
"id": 503500807,
"number": 1,
"head": {
"ref": "master",
"sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"repo": {
"id": 33884891,
"url": "https://api.github.com/repos/apache/airflow",
"name": "airflow"
}
},
"base": {
"ref": "master",
"sha": "0f02e45b7e127da3539752607ab02347d9fcd733",
"repo": {
"id": 169064440,
"url": "https://api.github.com/repos/TestingCodeReview/airflow",
"name": "airflow"
}
}
},
{
"url": "https://api.github.com/repos/mehrdad-shokri/incubator-airflow/pulls/2",
"id": 470767538,
"number": 2,
"head": {
"ref": "master",
"sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"repo": {
"id": 33884891,
"url": "https://api.github.com/repos/apache/airflow",
"name": "airflow"
}
},
"base": {
"ref": "master",
"sha": "5739ba28a7d8634ba3702d5fa39e5ef663f5bf7c",
"repo": {
"id": 110538040,
"url": "https://api.github.com/repos/mehrdad-shokri/incubator-airflow",
"name": "incubator-airflow"
}
}
}
],
"app": {
"id": 15368,
"slug": "github-actions",
"node_id": "MDM6QXBwMTUzNjg=",
"owner": {
"login": "github",
"id": 9919,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjk5MTk=",
"avatar_url": "https://avatars.githubusercontent.com/u/9919?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/github",
"html_url": "https://github.com/github",
"followers_url": "https://api.github.com/users/github/followers",
"following_url": "https://api.github.com/users/github/following{/other_user}",
"gists_url": "https://api.github.com/users/github/gists{/gist_id}",
"starred_url": "https://api.github.com/users/github/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/github/subscriptions",
"organizations_url": "https://api.github.com/users/github/orgs",
"repos_url": "https://api.github.com/users/github/repos",
"events_url": "https://api.github.com/users/github/events{/privacy}",
"received_events_url": "https://api.github.com/users/github/received_events",
"type": "Organization",
"site_admin": false
},
"name": "GitHub Actions",
"description": "Automate your workflow from idea to production",
"external_url": "https://help.github.com/en/actions",
"html_url": "https://github.com/apps/github-actions",
"created_at": "2018-07-30T09:30:17Z",
"updated_at": "2019-12-10T19:04:12Z",
"permissions": {
"actions": "write",
"checks": "write",
"contents": "write",
"deployments": "write",
"issues": "write",
"metadata": "read",
"organization_packages": "write",
"packages": "write",
"pages": "write",
"pull_requests": "write",
"repository_hooks": "write",
"repository_projects": "write",
"security_events": "write",
"statuses": "write",
"vulnerability_alerts": "read"
},
"events": [
"check_run",
"check_suite",
"create",
"delete",
"deployment",
"deployment_status",
"fork",
"gollum",
"issues",
"issue_comment",
"label",
"milestone",
"page_build",
"project",
"project_card",
"project_column",
"public",
"pull_request",
"pull_request_review",
"pull_request_review_comment",
"push",
"registry_package",
"release",
"repository",
"repository_dispatch",
"status",
"watch",
"workflow_dispatch",
"workflow_run"
]
},
"created_at": "2021-03-11T09:44:47Z",
"updated_at": "2021-03-11T10:09:45Z"
},
"app": {
"id": 15368,
"slug": "github-actions",
"node_id": "MDM6QXBwMTUzNjg=",
"owner": {
"login": "github",
"id": 9919,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjk5MTk=",
"avatar_url": "https://avatars.githubusercontent.com/u/9919?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/github",
"html_url": "https://github.com/github",
"followers_url": "https://api.github.com/users/github/followers",
"following_url": "https://api.github.com/users/github/following{/other_user}",
"gists_url": "https://api.github.com/users/github/gists{/gist_id}",
"starred_url": "https://api.github.com/users/github/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/github/subscriptions",
"organizations_url": "https://api.github.com/users/github/orgs",
"repos_url": "https://api.github.com/users/github/repos",
"events_url": "https://api.github.com/users/github/events{/privacy}",
"received_events_url": "https://api.github.com/users/github/received_events",
"type": "Organization",
"site_admin": false
},
"name": "GitHub Actions",
"description": "Automate your workflow from idea to production",
"external_url": "https://help.github.com/en/actions",
"html_url": "https://github.com/apps/github-actions",
"created_at": "2018-07-30T09:30:17Z",
"updated_at": "2019-12-10T19:04:12Z",
"permissions": {
"actions": "write",
"checks": "write",
"contents": "write",
"deployments": "write",
"issues": "write",
"metadata": "read",
"organization_packages": "write",
"packages": "write",
"pages": "write",
"pull_requests": "write",
"repository_hooks": "write",
"repository_projects": "write",
"security_events": "write",
"statuses": "write",
"vulnerability_alerts": "read"
},
"events": [
"check_run",
"check_suite",
"create",
"delete",
"deployment",
"deployment_status",
"fork",
"gollum",
"issues",
"issue_comment",
"label",
"milestone",
"page_build",
"project",
"project_card",
"project_column",
"public",
"pull_request",
"pull_request_review",
"pull_request_review_comment",
"push",
"registry_package",
"release",
"repository",
"repository_dispatch",
"status",
"watch",
"workflow_dispatch",
"workflow_run"
]
},
"pull_requests": [
{
"url": "https://api.github.com/repos/rpatil524/airflow/pulls/80",
"id": 581521861,
"number": 80,
"head": {
"ref": "master",
"sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"repo": {
"id": 33884891,
"url": "https://api.github.com/repos/apache/airflow",
"name": "airflow"
}
},
"base": {
"ref": "master",
"sha": "fff344443f764a0a5c9f5ba7dfbfa1e5e1115348",
"repo": {
"id": 336576614,
"url": "https://api.github.com/repos/rpatil524/airflow",
"name": "airflow"
}
}
},
{
"url": "https://api.github.com/repos/yufeixin/airflow/pulls/305",
"id": 581371796,
"number": 305,
"head": {
"ref": "master",
"sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"repo": {
"id": 33884891,
"url": "https://api.github.com/repos/apache/airflow",
"name": "airflow"
}
},
"base": {
"ref": "master",
"sha": "f9cc775adc0dff49b0e288d8f2745bf097017321",
"repo": {
"id": 320438482,
"url": "https://api.github.com/repos/yufeixin/airflow",
"name": "airflow"
}
}
},
{
"url": "https://api.github.com/repos/ddelange/airflow/pulls/6",
"id": 571967064,
"number": 6,
"head": {
"ref": "master",
"sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"repo": {
"id": 33884891,
"url": "https://api.github.com/repos/apache/airflow",
"name": "airflow"
}
},
"base": {
"ref": "master",
"sha": "f9c9e9c38f444a39987478f3d1a262db909de8c4",
"repo": {
"id": 219798743,
"url": "https://api.github.com/repos/ddelange/airflow",
"name": "airflow"
}
}
},
{
"url": "https://api.github.com/repos/weiplanet/airflow/pulls/1",
"id": 562187084,
"number": 1,
"head": {
"ref": "master",
"sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"repo": {
"id": 33884891,
"url": "https://api.github.com/repos/apache/airflow",
"name": "airflow"
}
},
"base": {
"ref": "master",
"sha": "2345cd1f0337f4f06a2f2595a1a59c30dc4a1535",
"repo": {
"id": 303579760,
"url": "https://api.github.com/repos/weiplanet/airflow",
"name": "airflow"
}
}
},
{
"url": "https://api.github.com/repos/TestingCodeReview/airflow/pulls/1",
"id": 503500807,
"number": 1,
"head": {
"ref": "master",
"sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"repo": {
"id": 33884891,
"url": "https://api.github.com/repos/apache/airflow",
"name": "airflow"
}
},
"base": {
"ref": "master",
"sha": "0f02e45b7e127da3539752607ab02347d9fcd733",
"repo": {
"id": 169064440,
"url": "https://api.github.com/repos/TestingCodeReview/airflow",
"name": "airflow"
}
}
},
{
"url": "https://api.github.com/repos/mehrdad-shokri/incubator-airflow/pulls/2",
"id": 470767538,
"number": 2,
"head": {
"ref": "master",
"sha": "9070ba280253e918c2582f52c2b77a07c311198f",
"repo": {
"id": 33884891,
"url": "https://api.github.com/repos/apache/airflow",
"name": "airflow"
}
},
"base": {
"ref": "master",
"sha": "5739ba28a7d8634ba3702d5fa39e5ef663f5bf7c",
"repo": {
"id": 110538040,
"url": "https://api.github.com/repos/mehrdad-shokri/incubator-airflow",
"name": "incubator-airflow"
}
}
}
]
},
"repository": {
"id": 33884891,
"node_id": "MDEwOlJlcG9zaXRvcnkzMzg4NDg5MQ==",
"name": "airflow",
"full_name": "apache/airflow",
"private": false,
"owner": {
"login": "apache",
"id": 47359,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjQ3MzU5",
"avatar_url": "https://avatars.githubusercontent.com/u/47359?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/apache",
"html_url": "https://github.com/apache",
"followers_url": "https://api.github.com/users/apache/followers",
"following_url": "https://api.github.com/users/apache/following{/other_user}",
"gists_url": "https://api.github.com/users/apache/gists{/gist_id}",
"starred_url": "https://api.github.com/users/apache/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/apache/subscriptions",
"organizations_url": "https://api.github.com/users/apache/orgs",
"repos_url": "https://api.github.com/users/apache/repos",
"events_url": "https://api.github.com/users/apache/events{/privacy}",
"received_events_url": "https://api.github.com/users/apache/received_events",
"type": "Organization",
"site_admin": false
},
"html_url": "https://github.com/apache/airflow",
"description": "Apache Airflow - A platform to programmatically author, schedule, and monitor workflows",
"fork": false,
"url": "https://api.github.com/repos/apache/airflow",
"forks_url": "https://api.github.com/repos/apache/airflow/forks",
"keys_url": "https://api.github.com/repos/apache/airflow/keys{/key_id}",
"collaborators_url": "https://api.github.com/repos/apache/airflow/collaborators{/collaborator}",
"teams_url": "https://api.github.com/repos/apache/airflow/teams",
"hooks_url": "https://api.github.com/repos/apache/airflow/hooks",
"issue_events_url": "https://api.github.com/repos/apache/airflow/issues/events{/number}",
"events_url": "https://api.github.com/repos/apache/airflow/events",
"assignees_url": "https://api.github.com/repos/apache/airflow/assignees{/user}",
"branches_url": "https://api.github.com/repos/apache/airflow/branches{/branch}",
"tags_url": "https://api.github.com/repos/apache/airflow/tags",
"blobs_url": "https://api.github.com/repos/apache/airflow/git/blobs{/sha}",
"git_tags_url": "https://api.github.com/repos/apache/airflow/git/tags{/sha}",
"git_refs_url": "https://api.github.com/repos/apache/airflow/git/refs{/sha}",
"trees_url": "https://api.github.com/repos/apache/airflow/git/trees{/sha}",
"statuses_url": "https://api.github.com/repos/apache/airflow/statuses/{sha}",
"languages_url": "https://api.github.com/repos/apache/airflow/languages",
"stargazers_url": "https://api.github.com/repos/apache/airflow/stargazers",
"contributors_url": "https://api.github.com/repos/apache/airflow/contributors",
"subscribers_url": "https://api.github.com/repos/apache/airflow/subscribers",
"subscription_url": "https://api.github.com/repos/apache/airflow/subscription",
"commits_url": "https://api.github.com/repos/apache/airflow/commits{/sha}",
"git_commits_url": "https://api.github.com/repos/apache/airflow/git/commits{/sha}",
"comments_url": "https://api.github.com/repos/apache/airflow/comments{/number}",
"issue_comment_url": "https://api.github.com/repos/apache/airflow/issues/comments{/number}",
"contents_url": "https://api.github.com/repos/apache/airflow/contents/{+path}",
"compare_url": "https://api.github.com/repos/apache/airflow/compare/{base}...{head}",
"merges_url": "https://api.github.com/repos/apache/airflow/merges",
"archive_url": "https://api.github.com/repos/apache/airflow/{archive_format}{/ref}",
"downloads_url": "https://api.github.com/repos/apache/airflow/downloads",
"issues_url": "https://api.github.com/repos/apache/airflow/issues{/number}",
"pulls_url": "https://api.github.com/repos/apache/airflow/pulls{/number}",
"milestones_url": "https://api.github.com/repos/apache/airflow/milestones{/number}",
"notifications_url": "https://api.github.com/repos/apache/airflow/notifications{?since,all,participating}",
"labels_url": "https://api.github.com/repos/apache/airflow/labels{/name}",
"releases_url": "https://api.github.com/repos/apache/airflow/releases{/id}",
"deployments_url": "https://api.github.com/repos/apache/airflow/deployments",
"created_at": "2015-04-13T18:04:58Z",
"updated_at": "2021-03-11T09:44:52Z",
"pushed_at": "2021-03-11T09:58:35Z",
"git_url": "git://github.com/apache/airflow.git",
"ssh_url": "git@github.com:apache/airflow.git",
"clone_url": "https://github.com/apache/airflow.git",
"svn_url": "https://github.com/apache/airflow",
"homepage": "https://airflow.apache.org/",
"size": 111044,
"stargazers_count": 20723,
"watchers_count": 20723,
"language": "Python",
"has_issues": true,
"has_projects": true,
"has_downloads": true,
"has_wiki": false,
"has_pages": false,
"forks_count": 8134,
"mirror_url": null,
"archived": false,
"disabled": false,
"open_issues_count": 987,
"license": {
"key": "apache-2.0",
"name": "Apache License 2.0",
"spdx_id": "Apache-2.0",
"url": "https://api.github.com/licenses/apache-2.0",
"node_id": "MDc6TGljZW5zZTI="
},
"forks": 8134,
"open_issues": 987,
"watchers": 20723,
"default_branch": "master"
},
"organization": {
"login": "apache",
"id": 47359,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjQ3MzU5",
"url": "https://api.github.com/orgs/apache",
"repos_url": "https://api.github.com/orgs/apache/repos",
"events_url": "https://api.github.com/orgs/apache/events",
"hooks_url": "https://api.github.com/orgs/apache/hooks",
"issues_url": "https://api.github.com/orgs/apache/issues",
"members_url": "https://api.github.com/orgs/apache/members{/member}",
"public_members_url": "https://api.github.com/orgs/apache/public_members{/member}",
"avatar_url": "https://avatars.githubusercontent.com/u/47359?v=4",
"description": ""
},
"sender": {
"login": "ashb",
"id": 34150,
"node_id": "MDQ6VXNlcjM0MTUw",
"avatar_url": "https://avatars.githubusercontent.com/u/34150?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/ashb",
"html_url": "https://github.com/ashb",
"followers_url": "https://api.github.com/users/ashb/followers",
"following_url": "https://api.github.com/users/ashb/following{/other_user}",
"gists_url": "https://api.github.com/users/ashb/gists{/gist_id}",
"starred_url": "https://api.github.com/users/ashb/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/ashb/subscriptions",
"organizations_url": "https://api.github.com/users/ashb/orgs",
"repos_url": "https://api.github.com/users/ashb/repos",
"events_url": "https://api.github.com/users/ashb/events{/privacy}",
"received_events_url": "https://api.github.com/users/ashb/received_events",
"type": "User",
"site_admin": false
},
"installation": {
"id": 14172953,
"node_id": "MDIzOkludGVncmF0aW9uSW5zdGFsbGF0aW9uMTQxNzI5NTM="
}
}</code></pre>
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org