You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/08/11 22:21:23 UTC
[GitHub] [beam] dannymartinm opened a new pull request, #22703: [GitHub Actions] Self-hosted runners migration
dannymartinm opened a new pull request, #22703:
URL: https://github.com/apache/beam/pull/22703
`DO NOT MERGE`
As part of the issue [21106](https://github.com/apache/beam/issues/21106), @elink21 and I have implemented the self-hosted runners for the Ubuntu and Windows operating systems.
In this PR you will find the following changes:
* Updates in workflows to `runs-on` self-hosted runners (Ubuntu and Windows)
* Docker and Kubernetes files to reproduce the environment of the Ubuntu runners in local and in Google Kubernetes Engine
* Scripts for the Windows runners implementation on GCP Compute Engine (VMs)
* Cloud Function scripts
* Detailed documentation in README files
Note: We will rely on GitHub-hosted runners for MacOS builds until a straightforward implementation approach comes out.
[BEAM-12812](https://issues.apache.org/jira/browse/BEAM-12812): Original Jira may contain additional context.
`Migration Release Plan`
Once this PR is approved, we are planning to create one PR per workflow and merge one by one in case some rollback is needed.
Thank you very much!
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on a diff in pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on code in PR #22703:
URL: https://github.com/apache/beam/pull/22703#discussion_r951796009
##########
.github/workflows/build_playground_backend.yml:
##########
@@ -17,16 +17,17 @@ name: Build And Deploy Playground Backend Application
on:
push:
- tags: 'v*'
+ tags: ['v*']
branches: ['master', 'release-*']
- pull_request:
+ pull_request_target:
paths: ['playground/backend/**']
branches: ['playground-staging']
workflow_dispatch:
+permissions: read-all
jobs:
build_playground_backend_docker_image:
name: Build Playground Backend App
- runs-on: ubuntu-latest
+ runs-on: [self-hosted, ubuntu-20.04]
Review Comment:
Sure, we created an ORG repo where we did a lot of testing in all the workflows over the past months and weeks.
The past week, we re-created the [ORG REPO](https://github.com/sre-apache-beam-dev/beam/actions) for showing you the behavior.
Currently, this branch `gh-actions-runners-migration` is already merge in `master` branch for testing our changes.
Please let us know if you have more questions regarding it, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1221023344
# [Codecov](https://codecov.io/gh/apache/beam/pull/22703?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#22703](https://codecov.io/gh/apache/beam/pull/22703?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7a469fd) into [master](https://codecov.io/gh/apache/beam/commit/00bbc72e005d741f632bb171b36d470f6544be55?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (00bbc72) will **decrease** coverage by `0.07%`.
> The diff coverage is `56.81%`.
> :exclamation: Current head 7a469fd differs from pull request most recent head b10baae. Consider uploading reports for the commit b10baae to get more accurate results
```diff
@@ Coverage Diff @@
## master #22703 +/- ##
==========================================
- Coverage 74.19% 74.11% -0.08%
==========================================
Files 708 712 +4
Lines 93475 93795 +320
==========================================
+ Hits 69352 69520 +168
- Misses 22848 22995 +147
- Partials 1275 1280 +5
```
| Flag | Coverage Δ | |
|---|---|---|
| go | `51.48% <46.95%> (-0.04%)` | :arrow_down: |
| python | `83.56% <71.06%> (-0.04%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/22703?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/graph/edge.go](https://codecov.io/gh/apache/beam/pull/22703/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2VkZ2UuZ28=) | `3.35% <ø> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/exec/translate.go](https://codecov.io/gh/apache/beam/pull/22703/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy90cmFuc2xhdGUuZ28=) | `13.92% <0.00%> (-0.37%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/graphx/serialize.go](https://codecov.io/gh/apache/beam/pull/22703/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3NlcmlhbGl6ZS5nbw==) | `27.34% <0.00%> (-0.17%)` | :arrow_down: |
| [...ks/python/apache\_beam/coders/coders\_test\_common.py](https://codecov.io/gh/apache/beam/pull/22703/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vY29kZXJzL2NvZGVyc190ZXN0X2NvbW1vbi5weQ==) | `98.54% <ø> (ø)` | |
| [...ache\_beam/runners/portability/local\_job\_service.py](https://codecov.io/gh/apache/beam/pull/22703/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9sb2NhbF9qb2Jfc2VydmljZS5weQ==) | `82.50% <0.00%> (ø)` | |
| [sdks/go/pkg/beam/pardo.go](https://codecov.io/gh/apache/beam/pull/22703/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9wYXJkby5nbw==) | `44.44% <10.00%> (-2.97%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/exec/fn.go](https://codecov.io/gh/apache/beam/pull/22703/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9mbi5nbw==) | `67.48% <21.42%> (-2.07%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/graphx/translate.go](https://codecov.io/gh/apache/beam/pull/22703/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3RyYW5zbGF0ZS5nbw==) | `41.58% <22.22%> (-0.88%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/funcx/fn.go](https://codecov.io/gh/apache/beam/pull/22703/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L2ZuLmdv) | `57.30% <31.25%> (-1.57%)` | :arrow_down: |
| [...s/inference/sklearn\_japanese\_housing\_regression.py](https://codecov.io/gh/apache/beam/pull/22703/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvaW5mZXJlbmNlL3NrbGVhcm5famFwYW5lc2VfaG91c2luZ19yZWdyZXNzaW9uLnB5) | `34.48% <34.48%> (ø)` | |
| ... and [24 more](https://codecov.io/gh/apache/beam/pull/22703/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1226596137
**Option 3:** Using `pull_request` with “**Require approval from first time contributors**”
- Pros
- Toil is reduced as only first time contributors are going to require manual approval.
- Repository security concerns are eliminated as write tokens are not granted to the `pull_request` directive.
- Cons
- Any person who is not a first time contributor can modify their workflows and the trigger events, then by opening a PR it will be executed without the requirement of approval, the untrusted code will run directly in our self-hosted runners.
**Ideal scenario:** A good scenario from our perspective would be to use `pull_request` and be able to lock the workflow to external modifications while still allowing to execute the safe ones from master without external approval.
* Unfortunately, **we haven’t found a way** with the options provided by GitHub to ensure the integrity of the workflow while still allowing executions from verified master jobs without external approval (Acting as a combination of the integrity feature of `pull_request_target` and repository security of `pull_request`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #22703:
URL: https://github.com/apache/beam/pull/22703#discussion_r951919156
##########
.github/ACTIONS.md:
##########
@@ -17,10 +17,78 @@
under the License.
-->
-> **PLEASE update this file if you add new github action or change name/trigger phrase of a github action.**
+> **PLEASE update this file if you add new GitHub Action or change name/trigger phrase of a GitHub Action.**
-## Beam Github Actions
+## About GitHub Actions Runners and Self-hosted Runners
+According to GitHub Docs, we can define a GitHub-hosted runner and a self-hosted runner as the following:
+* A [GitHub-hosted runner](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners) is a new virtual machine (VM) hosted by GitHub with the runner application and other tools preinstalled, and is available with Ubuntu Linux, Windows, or macOS operating systems.
+* A [self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) is a system that you deploy and manage to execute jobs from GitHub Actions on GitHub.com.
+## Apache Beam GitHub Actions
+
+Currently, we have both GitHub-hosted and self-hosted runners for running the GitHub Actions workflows, hosted on Google Cloud Platform(GCP) Virtual Machines and Google Kubernetes Engine(GKE). The majority of our workflows that run in Ubuntu and Windows run in self-hosted runners, except for those that runs on MacOS and the `Monitor Self-Hosted Runners Status` workflow that monitors our GCP self-hosted runners.
+
+### Getting Started with self-hosted runners
+* Refer to [this README](./gh-actions-self-hosted-runners/README.md) for the steps for creating your own self-hosted runners for testing your workflows.
+* Depending on your workflow's needs, it must specify the following `runs-on` tags to run in the specified operating system:
+ * Ubuntu 20.04 self-hosted runner: `[self-hosted, ubuntu-20.04]`
+ * Windows Server 2019 self-hosted runner: `[self-hosted,windows-server-2019]`
+ * MacOS GitHub-hosted runner: `macos-latest`
+* Every workflow that tests the source code, needs to have the workflow trigger `pull_request_target` instead of `pull_request`.
+* The workflow must have set read permissions for all the available scopes and jobs: `permissions: read-all`. It must be set at the top of the `jobs` directive.
+* For those workflows that have the `pull_request_target` trigger, in the checkout step must be added a ref to `${{ github.event.pull_request.head.sha }}`
+``` yaml
+ - name: Checkout code
+ uses: actions/checkout@v#
+ with:
+ ref: ${{ github.event.pull_request.head.sha }}
+```
+* If your workflow runs successfully in a GitHub-hosted runner but not in the self-hosted runner, it might need a new installation step.
+```yaml
+ - name: Setup Node
+ uses: actions/setup-node@v3
+ with:
+ node-version: 16
+```
+* You can find the GitHub-hosted runner installations in the following links:
+ * [Ubuntu-20.04](https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2004-Readme.md#installed-apt-packages)
+ * [Windows-2019](https://github.com/actions/runner-images/blob/main/images/win/Windows2019-Readme.md)
+
+#### GitHub Actions Example
+```yaml
+name: GitHub Actions Example
+on:
+ pull_request_target:
+ branches: ['master']
+permissions: read-all
+jobs:
+ github-actions-example:
+ runs-on: [self-hosted, ubuntu-20.04]
+ steps:
+ - name: Check out repository code
+ uses: actions/checkout@v2
+ with:
+ ref: ${{ github.event.pull_request.head.sha }}
+ - run: echo "This job is now running on a ubuntu server hosted by Apache Beam!"
+ - name: Setup Node
+ uses: actions/setup-node@v3
+ with:
+ node-version: 16
+ - name: Install npm dependencies
+ run: npm ci
+ working-directory: 'scripts/ci/your-path'
+ - name: Run Node.js code
+ run: npm run functionName
+ env:
+ VAR_1: my-var
+ working-directory: 'scripts/ci/your-path'
+```
+
+#### IMPORTANT for Committers
+* A **detailed review** for changes in the workflows is needed due to important **security concerns**.
+* **DO NOT** Approve and Run changes in the workflows in the PR Conversation tab, under "Workflow(s) awaiting approval".
Review Comment:
Is there a way to enforce this by some repo configuration or settings? Pinning Infra committer @damccorm
##########
.github/gh-actions-self-hosted-runners/helper-functions/generateToken/example.env:
##########
@@ -0,0 +1,8 @@
+# Please create a .env file with the corresponding values of the variables
Review Comment:
Can we deduplicate the three same example.env in sub directories, or just embed the example into documentation?
##########
.github/ACTIONS.md:
##########
@@ -17,10 +17,78 @@
under the License.
-->
-> **PLEASE update this file if you add new github action or change name/trigger phrase of a github action.**
+> **PLEASE update this file if you add new GitHub Action or change name/trigger phrase of a GitHub Action.**
-## Beam Github Actions
+## About GitHub Actions Runners and Self-hosted Runners
+According to GitHub Docs, we can define a GitHub-hosted runner and a self-hosted runner as the following:
+* A [GitHub-hosted runner](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners) is a new virtual machine (VM) hosted by GitHub with the runner application and other tools preinstalled, and is available with Ubuntu Linux, Windows, or macOS operating systems.
+* A [self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) is a system that you deploy and manage to execute jobs from GitHub Actions on GitHub.com.
+## Apache Beam GitHub Actions
+
+Currently, we have both GitHub-hosted and self-hosted runners for running the GitHub Actions workflows, hosted on Google Cloud Platform(GCP) Virtual Machines and Google Kubernetes Engine(GKE). The majority of our workflows that run in Ubuntu and Windows run in self-hosted runners, except for those that runs on MacOS and the `Monitor Self-Hosted Runners Status` workflow that monitors our GCP self-hosted runners.
+
+### Getting Started with self-hosted runners
+* Refer to [this README](./gh-actions-self-hosted-runners/README.md) for the steps for creating your own self-hosted runners for testing your workflows.
+* Depending on your workflow's needs, it must specify the following `runs-on` tags to run in the specified operating system:
+ * Ubuntu 20.04 self-hosted runner: `[self-hosted, ubuntu-20.04]`
+ * Windows Server 2019 self-hosted runner: `[self-hosted,windows-server-2019]`
+ * MacOS GitHub-hosted runner: `macos-latest`
+* Every workflow that tests the source code, needs to have the workflow trigger `pull_request_target` instead of `pull_request`.
Review Comment:
The GitHub documentation says "Avoid using this event if you need to build or run code from the pull request." What makes it necessary to change `pull_request` to `pull_request_target` trigger in existing workflows and does it have potential security concerns? I will also do some investigation about best practices.
##########
.github/workflows/build_wheels.yml:
##########
@@ -199,23 +202,26 @@ jobs:
run: gsutil cp -r -a public-read source/* ${{ env.GCP_PATH }}
build_wheels:
- name: Build python wheels on ${{matrix.arch}} for ${{ matrix.os_python.os }}
+ name: Build python wheels on ${{matrix.arch}} for ${{ join(matrix.os_python.os, ', ')}}
needs: build_source
env:
CIBW_ARCHS_LINUX: ${{matrix.arch}}
runs-on: ${{ matrix.os_python.os }}
strategy:
matrix:
os_python: [
- {"os": "ubuntu-latest", "python": "cp37-* cp38-* cp39-*"},
- {"os": "macos-latest", "python": "cp37-* cp38-* cp39-*"},
- {"os": "windows-latest", "python": "cp37-* cp38-* cp39-*"},
+ {"os": [self-hosted, ubuntu-20.04], "python": "cp37-* cp38-* cp39-*"},
+ {"os": [macos-latest], "python": "cp37-* cp38-* cp39-*"},
+ {"os": [self-hosted,windows-server-2019], "python": "cp37-* cp38-* cp39-*"},
Review Comment:
Is there a reason that there is a white space before 'ubuntu-20.02' but no white space before 'windows-server-2019' throughout the changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1353789026
> > @dannymartinm @elink21 what should we do with this PR at this point? The docs are probably still helpful to have, but the workflow changes have been pushed out into separate PRs
>
> Fortunately documentation was also divided into individual PRs, however due to the latest changes I will work on updating the infrastructure specs and it's documentation, if I'm not wrong this PR with all the changes was created as a reference for a quicker review, but everything was included in the other ones. For me it looks like we can close this one, what do you think @dannymartinm ?
Hi @elink21 and @damccorm , yes this PR was a previous step for the release and merging process just to have the full picture of our changes. We have split this PR into small ones (some of them already merged).
We can close this PR. Thank you team!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1212576667
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @Abacn for label build.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1241248704
Hi everyone,
In an offline discussion with @kerrydc and taking into account your approvals @damccorm @Abacn , we agreed to continue with Scenario 1.
We will proceed to create one PR per workflow and one for the self-hosted runners infrastructure.
Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1249290390
Reminder, please take a look at this pr: @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1230451240
This looks good to me. Currently we are experiencing 1h or more latency of GH Actions very often and it would be very helpful to get the self hosted runners deployed. Thanks for the work! Let us see @kennknowles 's input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1347013250
@dannymartinm @elink21 what should we do with this PR at this point? The docs are probably still helpful to have, but the workflow changes have been pushed out into separate PRs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #22703:
URL: https://github.com/apache/beam/pull/22703#discussion_r950281339
##########
.github/ACTIONS.md:
##########
@@ -17,10 +17,73 @@
under the License.
-->
-> **PLEASE update this file if you add new github action or change name/trigger phrase of a github action.**
+> **PLEASE update this file if you add new GitHub Action or change name/trigger phrase of a GitHub Action.**
-## Beam Github Actions
+## Beam GitHub Actions
+Currently, we have both GitHub-hosted and self-hosted (GCP cloud) runners for running the GitHub Actions workflows. The majority of our workflows that run in Ubuntu and Windows run in self-hosted runners, except for those that runs on MacOS and the `Monitor Self-Hosted Runners Status` workflow that monitors our GCP self-hosted runners.
+
+### Getting Started with self-hosted runners
+* The steps for creating your own self-hosted runners for testing your workflows in: `.github/gh-actions-self-hosted-runners/README.md`
+* Depending on your workflow's needs, it must specify the following `runs-on` tags to run in the specified operating system:
+ * Ubuntu 20.04 self-hosted runner: `[self-hosted, ubuntu-20.04]`
+ * Windows 2019 self-hosted runner: `[self-hosted,windows-server-2019]`
Review Comment:
```suggestion
* Windows Server 2019 self-hosted runner: `[self-hosted,windows-server-2019]`
```
##########
.github/ACTIONS.md:
##########
@@ -17,10 +17,73 @@
under the License.
-->
-> **PLEASE update this file if you add new github action or change name/trigger phrase of a github action.**
+> **PLEASE update this file if you add new GitHub Action or change name/trigger phrase of a GitHub Action.**
-## Beam Github Actions
+## Beam GitHub Actions
+Currently, we have both GitHub-hosted and self-hosted (GCP cloud) runners for running the GitHub Actions workflows. The majority of our workflows that run in Ubuntu and Windows run in self-hosted runners, except for those that runs on MacOS and the `Monitor Self-Hosted Runners Status` workflow that monitors our GCP self-hosted runners.
+
+### Getting Started with self-hosted runners
+* The steps for creating your own self-hosted runners for testing your workflows in: `.github/gh-actions-self-hosted-runners/README.md`
Review Comment:
```suggestion
* Refer to [this README](.github/gh-actions-self-hosted-runners/README.md) for the steps for creating your own self-hosted runners for testing your workflows.
```
##########
.github/workflows/build_playground_backend.yml:
##########
@@ -17,16 +17,17 @@ name: Build And Deploy Playground Backend Application
on:
push:
- tags: 'v*'
+ tags: ['v*']
branches: ['master', 'release-*']
- pull_request:
+ pull_request_target:
paths: ['playground/backend/**']
branches: ['playground-staging']
workflow_dispatch:
+permissions: read-all
jobs:
build_playground_backend_docker_image:
name: Build Playground Backend App
- runs-on: ubuntu-latest
+ runs-on: [self-hosted, ubuntu-20.04]
Review Comment:
A generic question: is there a fork repository showing this migration is working?
##########
.github/ACTIONS.md:
##########
@@ -17,10 +17,73 @@
under the License.
-->
-> **PLEASE update this file if you add new github action or change name/trigger phrase of a github action.**
+> **PLEASE update this file if you add new GitHub Action or change name/trigger phrase of a GitHub Action.**
-## Beam Github Actions
+## Beam GitHub Actions
+Currently, we have both GitHub-hosted and self-hosted (GCP cloud) runners for running the GitHub Actions workflows. The majority of our workflows that run in Ubuntu and Windows run in self-hosted runners, except for those that runs on MacOS and the `Monitor Self-Hosted Runners Status` workflow that monitors our GCP self-hosted runners.
Review Comment:
Unfortunately the usage of runner in term "GitHub Runner" conflicts with Beam Runner. May need some clarification at the beginning of the document.
##########
.github/gh-actions-self-hosted-runners/README.md:
##########
@@ -0,0 +1,106 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+# GitHub Actions - Self-hosted Runners
+The current GitHub Actions workflows are being tested on multiple operating systems, such as Ubuntu, Windows and MacOS. The way to migrate these runners from GitHub to GCP is by implementing self-hosted runners, so we implemented them in both Ubuntu and Windows environments, going with Google Kubernetes Engine and Google Cloud Compute VMs instances respectively.
+
+On the other hand, we will rely on GitHub-hosted runners for MacOS builds until a straightforward implementation approach comes out.
+
+## Ubuntu
+Ubuntu Self-hosted runners are stored in Artifact Registry and implemented using Google Kubernetes Engine with the following specifications:
+
+#### Cluster
+* Cluster: [gh-actions-linux-runners](https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/gh-actions-linux-runners/details?project=apache-beam-testing)
+* Image: [linux-github-actions-runner](https://console.cloud.google.com/artifacts/docker/apache-beam-testing/us-central1/beam-github-actions/linux-github-actions-runner?project=apache-beam-testing)
+
+#### Pool
+* Number of nodes: 5
+* Cluster Autoscaler: ON
+ * Minimum number of nodes: 5
+ * Maximum number of nodes: 10
+
+#### Node
+* Machine Type: e2-custom-6-18432
+* Disk Size: 100 GB
+* CPU: 6 vCPUs
+* Memory : 18 GB
+
+#### Pod
+* Image: $LOCAL_IMAGE_NAME LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest
+* CPU: 2
+* Memory: 1028 Mi
+* Volumes
+ * docker-sock
+ * gcloud-key
+* Secret env variables
+ * Kubernetes Secrets
+ * github-actions-secrets
+ * gcloud-key
+
+#### AutoScaling
+* Horizontal Pod Autoscaling
+ * 5-10 nodes (From Pool Cluster Autoscaler)
+ * HorizontalPodAutoscaler
+ * Min replicas: 10
+ * Max replicas: 20
+ * CPU utilization: 70%
+* Vertical Pod Autoscaling
+ * updateMode: "Auto"
+
+
+## Windows
+Windows Virtual machines have the following specifications
+
+#### VM specifications
+* Instance Template: _TODO: Add GCP link when final resources are in place._
+* Machine Type: n2-standard-2
+* Disk Size: 70 GB
+* Disk Image: [disk-image-windows-runner](https://console.cloud.google.com/compute/imagesDetail/projects/apache-beam-testing/global/images/disk-image-windows-runner?project=apache-beam-testing)
+* CPU: 2 vCPUs
+* Memory : 8 GB
+
+#### Instance group settings
+* Region: us-west1 (multizone)
+* Scale-out metric: 70% of CPU Usage.
+* Cooldown period: 300s
+
+#### Notes:
+At first glance we considered implementing Windows runners using K8s, however this was not optimal because of the following reasons:
+
+* VS Build tools are required for certain workflows, unfortunately official images that support this dependency are huge in size, reaching 20GB easily which is not an ideal case for k8S management.
+* Windows Subsystem For Linux(WSL) is a feature that allows to execute bash scripts inside Windows which removes tech debt by avoiding writing steps in powershell, but this feature is disabled with payload removed in Windows containers.
Review Comment:
Does this still invoke win32 applications? For example, for python unit test workflow, the script (https://github.com/apache/beam/blob/7a469fd20ef198a38e1df6af081062904dd1cbbb/sdks/python/scripts/run_pytest.sh#L39) invokes pytest should still be win32 version.
##########
.github/gh-actions-self-hosted-runners/self-hosted-windows/README.md:
##########
@@ -0,0 +1,54 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# GitHub Actions - Self-hosted Windows Runners
+
+This folder contains the required resources to deploy the GitHub Actions self-hosted runners for the workflows running in Windows OS.
+
+#### How to build a new instance template for the instance group?
+
+* Create a new VM Instance using Windows 2019 Datacenter OS with at least 70GB of disk, 2VCPUs and 8 GB of RAM.
Review Comment:
```suggestion
* Create a new VM Instance using Windows 2019 Datacenter OS with at least 70GB of disk, 2vCPUs and 8 GB of RAM.
```
##########
.github/ACTIONS.md:
##########
@@ -17,10 +17,73 @@
under the License.
-->
-> **PLEASE update this file if you add new github action or change name/trigger phrase of a github action.**
+> **PLEASE update this file if you add new GitHub Action or change name/trigger phrase of a GitHub Action.**
-## Beam Github Actions
+## Beam GitHub Actions
+Currently, we have both GitHub-hosted and self-hosted (GCP cloud) runners for running the GitHub Actions workflows. The majority of our workflows that run in Ubuntu and Windows run in self-hosted runners, except for those that runs on MacOS and the `Monitor Self-Hosted Runners Status` workflow that monitors our GCP self-hosted runners.
Review Comment:
```suggestion
Currently, we have both GitHub-hosted and self-hosted (GCP cloud) runners (virtual machines) for running the GitHub Actions workflows. The majority of our workflows that run in Ubuntu and Windows run in self-hosted runners, except for those that runs on MacOS and the `Monitor Self-Hosted Runners Status` workflow that monitors our GCP self-hosted runners.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on a diff in pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on code in PR #22703:
URL: https://github.com/apache/beam/pull/22703#discussion_r951800794
##########
.github/gh-actions-self-hosted-runners/README.md:
##########
@@ -0,0 +1,106 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+# GitHub Actions - Self-hosted Runners
+The current GitHub Actions workflows are being tested on multiple operating systems, such as Ubuntu, Windows and MacOS. The way to migrate these runners from GitHub to GCP is by implementing self-hosted runners, so we implemented them in both Ubuntu and Windows environments, going with Google Kubernetes Engine and Google Cloud Compute VMs instances respectively.
+
+On the other hand, we will rely on GitHub-hosted runners for MacOS builds until a straightforward implementation approach comes out.
+
+## Ubuntu
+Ubuntu Self-hosted runners are stored in Artifact Registry and implemented using Google Kubernetes Engine with the following specifications:
+
+#### Cluster
+* Cluster: [gh-actions-linux-runners](https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/gh-actions-linux-runners/details?project=apache-beam-testing)
+* Image: [linux-github-actions-runner](https://console.cloud.google.com/artifacts/docker/apache-beam-testing/us-central1/beam-github-actions/linux-github-actions-runner?project=apache-beam-testing)
+
+#### Pool
+* Number of nodes: 5
+* Cluster Autoscaler: ON
+ * Minimum number of nodes: 5
+ * Maximum number of nodes: 10
+
+#### Node
+* Machine Type: e2-custom-6-18432
+* Disk Size: 100 GB
+* CPU: 6 vCPUs
+* Memory : 18 GB
+
+#### Pod
+* Image: $LOCAL_IMAGE_NAME LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest
+* CPU: 2
+* Memory: 1028 Mi
+* Volumes
+ * docker-sock
+ * gcloud-key
+* Secret env variables
+ * Kubernetes Secrets
+ * github-actions-secrets
+ * gcloud-key
+
+#### AutoScaling
+* Horizontal Pod Autoscaling
+ * 5-10 nodes (From Pool Cluster Autoscaler)
+ * HorizontalPodAutoscaler
+ * Min replicas: 10
+ * Max replicas: 20
+ * CPU utilization: 70%
+* Vertical Pod Autoscaling
+ * updateMode: "Auto"
+
+
+## Windows
+Windows Virtual machines have the following specifications
+
+#### VM specifications
+* Instance Template: _TODO: Add GCP link when final resources are in place._
+* Machine Type: n2-standard-2
+* Disk Size: 70 GB
+* Disk Image: [disk-image-windows-runner](https://console.cloud.google.com/compute/imagesDetail/projects/apache-beam-testing/global/images/disk-image-windows-runner?project=apache-beam-testing)
+* CPU: 2 vCPUs
+* Memory : 8 GB
+
+#### Instance group settings
+* Region: us-west1 (multizone)
+* Scale-out metric: 70% of CPU Usage.
+* Cooldown period: 300s
+
+#### Notes:
+At first glance we considered implementing Windows runners using K8s, however this was not optimal because of the following reasons:
+
+* VS Build tools are required for certain workflows, unfortunately official images that support this dependency are huge in size, reaching 20GB easily which is not an ideal case for k8S management.
+* Windows Subsystem For Linux(WSL) is a feature that allows to execute bash scripts inside Windows which removes tech debt by avoiding writing steps in powershell, but this feature is disabled with payload removed in Windows containers.
Review Comment:
Yes, it includes the win32 tests as well. You can find the examples in [Python tests](https://github.com/sre-apache-beam-dev/beam/runs/7957326225?check_suite_focus=true ) workflow in the `Run tests basic windows` step.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1227349339
Thanks @elink21 and @dannymartinm for laying out the options. @kennknowles this would be a good one to get your input on.
I think we should probably ignore scenario 3. That's not a real security boundary and is incredibly easy to circumvent (submit a small doc improvement or more general small fix, then do something malicious). So then the remaining question: is scenario 1 actually problematic. I'd argue that its not, or at least its not a regression from our current security standards because right now the same security risk (someone changes code that has access to secrets) exists in Jenkins.
Scenario 2 is a tightening of our current security boundarys that IMO imposes too large of a burden on committers to be worth it. With that said, it does mean that we need to be thoughtful about how/when we pass our secrets into our code and how much risk we're willing to take on (in particular we should be tight about how we scope our GitHub tokens so they don't have arbitrary repo read/write power).
I'd vote for scenario 1 here, with the acknowledgement that it comes with some risk if we're not careful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on a diff in pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on code in PR #22703:
URL: https://github.com/apache/beam/pull/22703#discussion_r954383527
##########
.github/gh-actions-self-hosted-runners/helper-functions/generateToken/example.env:
##########
@@ -0,0 +1,8 @@
+# Please create a .env file with the corresponding values of the variables
Review Comment:
Deduplicated the `example.env` file and included the helper functions details in a new README. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1234812561
> Hey @dannymartinm , just 3 days ago we added a new "Tour of Beam Go tests" action for our Tour of Beam work (please see https://github.com/apache/beam/pull/22556/files#diff-3102b5053f618d3d8d89c16bb6a4bfa107674d948bb5e936035242d5fbf7db3d ). Do we need to update to self-host runner as a separate issue after this PR is merged?
Hi @olehborysevych ! Yes, that would be great. We will let you know once the self-hosted runners are in place.
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
elink21 commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1347254146
> @dannymartinm @elink21 what should we do with this PR at this point? The docs are probably still helpful to have, but the workflow changes have been pushed out into separate PRs
Fortunately documentation was also divided into individual PRs, however due to the latest changes I will work on updating the infrastructure specs and it's documentation, if I'm not wrong this PR with all the changes was created as a reference for a quicker review, but everything was included in the other ones. For me it looks like we can close this one, what do you think @dannymartinm ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm closed pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
damccorm closed pull request #22703: [GitHub Actions] Self-hosted runners migration
URL: https://github.com/apache/beam/pull/22703
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #22703:
URL: https://github.com/apache/beam/pull/22703#discussion_r952716785
##########
.github/ACTIONS.md:
##########
@@ -17,10 +17,78 @@
under the License.
-->
-> **PLEASE update this file if you add new github action or change name/trigger phrase of a github action.**
+> **PLEASE update this file if you add new GitHub Action or change name/trigger phrase of a GitHub Action.**
-## Beam Github Actions
+## About GitHub Actions Runners and Self-hosted Runners
+According to GitHub Docs, we can define a GitHub-hosted runner and a self-hosted runner as the following:
+* A [GitHub-hosted runner](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners) is a new virtual machine (VM) hosted by GitHub with the runner application and other tools preinstalled, and is available with Ubuntu Linux, Windows, or macOS operating systems.
+* A [self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) is a system that you deploy and manage to execute jobs from GitHub Actions on GitHub.com.
+## Apache Beam GitHub Actions
+
+Currently, we have both GitHub-hosted and self-hosted runners for running the GitHub Actions workflows, hosted on Google Cloud Platform(GCP) Virtual Machines and Google Kubernetes Engine(GKE). The majority of our workflows that run in Ubuntu and Windows run in self-hosted runners, except for those that runs on MacOS and the `Monitor Self-Hosted Runners Status` workflow that monitors our GCP self-hosted runners.
+
+### Getting Started with self-hosted runners
+* Refer to [this README](./gh-actions-self-hosted-runners/README.md) for the steps for creating your own self-hosted runners for testing your workflows.
+* Depending on your workflow's needs, it must specify the following `runs-on` tags to run in the specified operating system:
+ * Ubuntu 20.04 self-hosted runner: `[self-hosted, ubuntu-20.04]`
+ * Windows Server 2019 self-hosted runner: `[self-hosted,windows-server-2019]`
+ * MacOS GitHub-hosted runner: `macos-latest`
+* Every workflow that tests the source code, needs to have the workflow trigger `pull_request_target` instead of `pull_request`.
+* The workflow must have set read permissions for all the available scopes and jobs: `permissions: read-all`. It must be set at the top of the `jobs` directive.
+* For those workflows that have the `pull_request_target` trigger, in the checkout step must be added a ref to `${{ github.event.pull_request.head.sha }}`
+``` yaml
+ - name: Checkout code
+ uses: actions/checkout@v#
+ with:
+ ref: ${{ github.event.pull_request.head.sha }}
+```
+* If your workflow runs successfully in a GitHub-hosted runner but not in the self-hosted runner, it might need a new installation step.
+```yaml
+ - name: Setup Node
+ uses: actions/setup-node@v3
+ with:
+ node-version: 16
+```
+* You can find the GitHub-hosted runner installations in the following links:
+ * [Ubuntu-20.04](https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2004-Readme.md#installed-apt-packages)
+ * [Windows-2019](https://github.com/actions/runner-images/blob/main/images/win/Windows2019-Readme.md)
+
+#### GitHub Actions Example
+```yaml
+name: GitHub Actions Example
+on:
+ pull_request_target:
+ branches: ['master']
+permissions: read-all
+jobs:
+ github-actions-example:
+ runs-on: [self-hosted, ubuntu-20.04]
+ steps:
+ - name: Check out repository code
+ uses: actions/checkout@v2
+ with:
+ ref: ${{ github.event.pull_request.head.sha }}
+ - run: echo "This job is now running on a ubuntu server hosted by Apache Beam!"
+ - name: Setup Node
+ uses: actions/setup-node@v3
+ with:
+ node-version: 16
+ - name: Install npm dependencies
+ run: npm ci
+ working-directory: 'scripts/ci/your-path'
+ - name: Run Node.js code
+ run: npm run functionName
+ env:
+ VAR_1: my-var
+ working-directory: 'scripts/ci/your-path'
+```
+
+#### IMPORTANT for Committers
+* A **detailed review** for changes in the workflows is needed due to important **security concerns**.
+* **DO NOT** Approve and Run changes in the workflows in the PR Conversation tab, under "Workflow(s) awaiting approval".
Review Comment:
I don't think so. Honestly, I'm not really sure that this security boundary is enforceable; once someone has submitted a contribution they will be allowlisted for actions on their future runs I believe (unless we've changed that setting, but doing so is painful).
@dannymartinm what are the consequences of someone crossing this security boundary? Would they have admin privileges on the repo, the ability to takeover our runners, or something else? I guess probably they could exfiltrate secrets
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kerrydc commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
kerrydc commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1232081630
I agree with @damccorm that scenario 1 is best. Overall no issues with this PR, thanks for the hard work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on a diff in pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on code in PR #22703:
URL: https://github.com/apache/beam/pull/22703#discussion_r954377638
##########
.github/ACTIONS.md:
##########
@@ -17,10 +17,78 @@
under the License.
-->
-> **PLEASE update this file if you add new github action or change name/trigger phrase of a github action.**
+> **PLEASE update this file if you add new GitHub Action or change name/trigger phrase of a GitHub Action.**
-## Beam Github Actions
+## About GitHub Actions Runners and Self-hosted Runners
+According to GitHub Docs, we can define a GitHub-hosted runner and a self-hosted runner as the following:
+* A [GitHub-hosted runner](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners) is a new virtual machine (VM) hosted by GitHub with the runner application and other tools preinstalled, and is available with Ubuntu Linux, Windows, or macOS operating systems.
+* A [self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) is a system that you deploy and manage to execute jobs from GitHub Actions on GitHub.com.
+## Apache Beam GitHub Actions
+
+Currently, we have both GitHub-hosted and self-hosted runners for running the GitHub Actions workflows, hosted on Google Cloud Platform(GCP) Virtual Machines and Google Kubernetes Engine(GKE). The majority of our workflows that run in Ubuntu and Windows run in self-hosted runners, except for those that runs on MacOS and the `Monitor Self-Hosted Runners Status` workflow that monitors our GCP self-hosted runners.
+
+### Getting Started with self-hosted runners
+* Refer to [this README](./gh-actions-self-hosted-runners/README.md) for the steps for creating your own self-hosted runners for testing your workflows.
+* Depending on your workflow's needs, it must specify the following `runs-on` tags to run in the specified operating system:
+ * Ubuntu 20.04 self-hosted runner: `[self-hosted, ubuntu-20.04]`
+ * Windows Server 2019 self-hosted runner: `[self-hosted,windows-server-2019]`
+ * MacOS GitHub-hosted runner: `macos-latest`
+* Every workflow that tests the source code, needs to have the workflow trigger `pull_request_target` instead of `pull_request`.
Review Comment:
We were concerned about using `pull_request` because any person can modify the workflows(inject insecure code) and run it without any prior review/approval.
So far we have tested and found different implementation options, which have pros and cons.
We have added the details in the PR conversation. Please feel free to add your comments and questions.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
elink21 commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1226589948
Below we will describe the different possible scenarios we have found for the Github actions features/restrictions and repo configurations. Please feel free to add your comments and questions.
**Scenario 1**: `pull_request_target` for workflows with “**Read repository contents permissions**“ and “**Require approval from all outside collaborators**” at Beam repo level
- Pros
- Avoid updating and injecting insecure code to the workflow as it will always run the master workflow
- Workflows must have the checkout reference to the PR in order to run the changes in the code (outside of .github/workflows/*)
- Seed job will be emulated using the Approve and Run built in feature of GH Actions and updating the event trigger to `pull_request;` no modifications would be allowed either for new or existent jobs unless a committer explicitly approves the job from UI.
- The “Read repository contents permissions” option will give ONLY the content and metadata read permissions by default. If the workflow needs another permission, it will need to be explicitly specified.
- Workflows with `pull_request_target` directive are always executed in the master context, and can’t be modified externally unless someone with admin rights exposes secrets directly inside the master workflow (as environment variables i.e), those secrets shouldn’t be accessible from the workflow executions.
- Cons
- Repository Secrets can be accessed in the src code if the secret is explicitly passed to it as an environment variable. Committers/reviewers must reject passing secrets to the code if the workflow is vulnerable to modifications or can be triggered by unsafe events.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
olehborysevych commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1234109086
Hey @dannymartinm , just 3 days ago we added a new "Tour of Beam Go tests" action for our Tour of Beam work (please see https://github.com/apache/beam/pull/22556/files#diff-3102b5053f618d3d8d89c16bb6a4bfa107674d948bb5e936035242d5fbf7db3d ). Do we need to update to self-host runner as a separate issue after this PR is merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
elink21 commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1308952711
Status update: reviewed by @damccorm, waiting for the migration window to deploy the infrastructure and request the merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on a diff in pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on code in PR #22703:
URL: https://github.com/apache/beam/pull/22703#discussion_r954379592
##########
.github/ACTIONS.md:
##########
@@ -17,10 +17,78 @@
under the License.
-->
-> **PLEASE update this file if you add new github action or change name/trigger phrase of a github action.**
+> **PLEASE update this file if you add new GitHub Action or change name/trigger phrase of a GitHub Action.**
-## Beam Github Actions
+## About GitHub Actions Runners and Self-hosted Runners
+According to GitHub Docs, we can define a GitHub-hosted runner and a self-hosted runner as the following:
+* A [GitHub-hosted runner](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners) is a new virtual machine (VM) hosted by GitHub with the runner application and other tools preinstalled, and is available with Ubuntu Linux, Windows, or macOS operating systems.
+* A [self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners) is a system that you deploy and manage to execute jobs from GitHub Actions on GitHub.com.
+## Apache Beam GitHub Actions
+
+Currently, we have both GitHub-hosted and self-hosted runners for running the GitHub Actions workflows, hosted on Google Cloud Platform(GCP) Virtual Machines and Google Kubernetes Engine(GKE). The majority of our workflows that run in Ubuntu and Windows run in self-hosted runners, except for those that runs on MacOS and the `Monitor Self-Hosted Runners Status` workflow that monitors our GCP self-hosted runners.
+
+### Getting Started with self-hosted runners
+* Refer to [this README](./gh-actions-self-hosted-runners/README.md) for the steps for creating your own self-hosted runners for testing your workflows.
+* Depending on your workflow's needs, it must specify the following `runs-on` tags to run in the specified operating system:
+ * Ubuntu 20.04 self-hosted runner: `[self-hosted, ubuntu-20.04]`
+ * Windows Server 2019 self-hosted runner: `[self-hosted,windows-server-2019]`
+ * MacOS GitHub-hosted runner: `macos-latest`
+* Every workflow that tests the source code, needs to have the workflow trigger `pull_request_target` instead of `pull_request`.
+* The workflow must have set read permissions for all the available scopes and jobs: `permissions: read-all`. It must be set at the top of the `jobs` directive.
+* For those workflows that have the `pull_request_target` trigger, in the checkout step must be added a ref to `${{ github.event.pull_request.head.sha }}`
+``` yaml
+ - name: Checkout code
+ uses: actions/checkout@v#
+ with:
+ ref: ${{ github.event.pull_request.head.sha }}
+```
+* If your workflow runs successfully in a GitHub-hosted runner but not in the self-hosted runner, it might need a new installation step.
+```yaml
+ - name: Setup Node
+ uses: actions/setup-node@v3
+ with:
+ node-version: 16
+```
+* You can find the GitHub-hosted runner installations in the following links:
+ * [Ubuntu-20.04](https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2004-Readme.md#installed-apt-packages)
+ * [Windows-2019](https://github.com/actions/runner-images/blob/main/images/win/Windows2019-Readme.md)
+
+#### GitHub Actions Example
+```yaml
+name: GitHub Actions Example
+on:
+ pull_request_target:
+ branches: ['master']
+permissions: read-all
+jobs:
+ github-actions-example:
+ runs-on: [self-hosted, ubuntu-20.04]
+ steps:
+ - name: Check out repository code
+ uses: actions/checkout@v2
+ with:
+ ref: ${{ github.event.pull_request.head.sha }}
+ - run: echo "This job is now running on a ubuntu server hosted by Apache Beam!"
+ - name: Setup Node
+ uses: actions/setup-node@v3
+ with:
+ node-version: 16
+ - name: Install npm dependencies
+ run: npm ci
+ working-directory: 'scripts/ci/your-path'
+ - name: Run Node.js code
+ run: npm run functionName
+ env:
+ VAR_1: my-var
+ working-directory: 'scripts/ci/your-path'
+```
+
+#### IMPORTANT for Committers
+* A **detailed review** for changes in the workflows is needed due to important **security concerns**.
+* **DO NOT** Approve and Run changes in the workflows in the PR Conversation tab, under "Workflow(s) awaiting approval".
Review Comment:
Hi @damccorm , we have tested and found different implementation options, which have pros and cons related to security.
We added the details in this PR conversation. Please feel free to add your comments and questions.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on a diff in pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on code in PR #22703:
URL: https://github.com/apache/beam/pull/22703#discussion_r954382879
##########
.github/workflows/build_wheels.yml:
##########
@@ -199,23 +202,26 @@ jobs:
run: gsutil cp -r -a public-read source/* ${{ env.GCP_PATH }}
build_wheels:
- name: Build python wheels on ${{matrix.arch}} for ${{ matrix.os_python.os }}
+ name: Build python wheels on ${{matrix.arch}} for ${{ join(matrix.os_python.os, ', ')}}
needs: build_source
env:
CIBW_ARCHS_LINUX: ${{matrix.arch}}
runs-on: ${{ matrix.os_python.os }}
strategy:
matrix:
os_python: [
- {"os": "ubuntu-latest", "python": "cp37-* cp38-* cp39-*"},
- {"os": "macos-latest", "python": "cp37-* cp38-* cp39-*"},
- {"os": "windows-latest", "python": "cp37-* cp38-* cp39-*"},
+ {"os": [self-hosted, ubuntu-20.04], "python": "cp37-* cp38-* cp39-*"},
+ {"os": [macos-latest], "python": "cp37-* cp38-* cp39-*"},
+ {"os": [self-hosted,windows-server-2019], "python": "cp37-* cp38-* cp39-*"},
Review Comment:
I think it was a typo, it's resolved now. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1227452678
Thanks @elink21 for the detailed information. I agree that scenario 1 looks most preferable, and it matches the current Jenkins workflow most.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on a diff in pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on code in PR #22703:
URL: https://github.com/apache/beam/pull/22703#discussion_r951789086
##########
.github/ACTIONS.md:
##########
@@ -17,10 +17,73 @@
under the License.
-->
-> **PLEASE update this file if you add new github action or change name/trigger phrase of a github action.**
+> **PLEASE update this file if you add new GitHub Action or change name/trigger phrase of a GitHub Action.**
-## Beam Github Actions
+## Beam GitHub Actions
+Currently, we have both GitHub-hosted and self-hosted (GCP cloud) runners for running the GitHub Actions workflows. The majority of our workflows that run in Ubuntu and Windows run in self-hosted runners, except for those that runs on MacOS and the `Monitor Self-Hosted Runners Status` workflow that monitors our GCP self-hosted runners.
Review Comment:
A new section `About GitHub Actions Runners and Self-hosted Runners` is added. It includes the GitHub definition of both `GitHub-hosted` and `Self-hosted` runners for more clarification. Please let us know if it's fine or if more details need to be added. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #22703: [GitHub Actions] Self-hosted runners migration
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #22703:
URL: https://github.com/apache/beam/pull/22703#issuecomment-1219937145
Thanks for the contribution! Looking into it, will need some time..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org