You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/09 20:38:03 UTC
[GitHub] [beam] dannymartinm opened a new pull request, #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
dannymartinm opened a new pull request, #23158:
URL: https://github.com/apache/beam/pull/23158
As part of the issue https://github.com/apache/beam/issues/21106, @elink21 and I have implemented the self-hosted runners for the Ubuntu and Windows operating systems.
Please see the Approved Pull Request ([PR#22703](https://github.com/apache/beam/pull/22703)) for reference.
In this PR you will find the following changes:
* Docker and Kubernetes files to reproduce the environment of the Ubuntu runners in local and in Google Kubernetes Engine
* Scripts for the Windows runners implementation on GCP Compute Engine (VMs)
* Cloud Function scripts
* Detailed documentation in README files
Note: We will rely on GitHub-hosted runners for MacOS builds until a straightforward implementation approach comes out.
[PR#16511 ](https://github.com/apache/beam/pull/16511): Previous PR for reference.
[BEAM-12812](https://issues.apache.org/jira/browse/BEAM-12812): Original Jira may contain additional context.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r969982899
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/index.js:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function provides registration tokens for windows and linux self-hosted
+//runners, a service account with the appropriated permissions should be used to
+// invoke the Cloud Function.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+
+async function getRunnerToken() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_NAME,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let access = await octokit.request(`POST /app/installations/${process.env.APP_INSTALLATION_ID}}/access_tokens`, {
+ repositories: [
+ 'beam'
+ ],
+ permissions: {
+ organization_self_hosted_runners: "write",
+ }
+ });
+ //In order to access the registration token endpoint, an additional Auth token must be used
+ let authToken = access.data.token;
+ let auth = " token " + authToken;
Review Comment:
Nit, prefer:
```suggestion
let auth = ` token ${authToken}`;
```
##########
.github/gh-actions-self-hosted-runners/README.md:
##########
@@ -0,0 +1,121 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+# GitHub Actions - Self-hosted Runners
+The current GitHub Actions workflows are being tested on multiple operating systems, such as Ubuntu, Windows and MacOS. The way to migrate these runners from GitHub to GCP is by implementing self-hosted runners, so we implemented them in both Ubuntu and Windows environments, going with Google Kubernetes Engine and Google Cloud Compute VMs instances respectively.
+
+On the other hand, we will rely on GitHub-hosted runners for MacOS builds until a straightforward implementation approach comes out.
+
+## Ubuntu
+Ubuntu Self-hosted runners are stored in Artifact Registry and implemented using Google Kubernetes Engine with the following specifications:
+
+#### Cluster
+* Cluster: [gh-actions-linux-runners](https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/gh-actions-linux-runners/details?project=apache-beam-testing)
+* Image: [linux-github-actions-runner](https://console.cloud.google.com/artifacts/docker/apache-beam-testing/us-central1/beam-github-actions/linux-github-actions-runner?project=apache-beam-testing)
+
+#### Pool
+* Number of nodes: 5
+* Cluster Autoscaler: ON
+ * Minimum number of nodes: 5
+ * Maximum number of nodes: 10
+
+#### Node
+* Machine Type: e2-custom-6-18432
+* Disk Size: 100 GB
+* CPU: 6 vCPUs
+* Memory : 18 GB
+
+#### Pod
+##### Container 1: gh-actions-runner
+* Image: $LOCAL_IMAGE_NAME LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest
+* CPU: 2
+* Memory: 1028 Mi
+* Volumes
+ * gcloud-key
+ * docker-certs-client
+* Environment variables
+ * Container variables
+ * GOOGLE_APPLICATION_CREDENTIALS
+ * DOCKER_HOST
+ * DOCKER_TLS_VERIFY
+ * DOCKER_CERT_PATH
+ * Kubernetes secret env variables
+ * github-actions-secrets
+ * gcloud-key
+
+##### Container 2: dind
+* Image: `docker:20.10.17-dind`
+* Volumes
+ * dind-storage
+ * docker-certs-client
+
+###### Pod Diagram
+
+
+#### AutoScaling
+* Horizontal Pod Autoscaling
+ * 5-10 nodes (From Pool Cluster Autoscaler)
+ * HorizontalPodAutoscaler
+ * Min replicas: 10
+ * Max replicas: 20
+ * CPU utilization: 70%
+* Vertical Pod Autoscaling
+ * updateMode: "Auto"
+
+
+## Windows
+Windows Virtual machines have the following specifications
+
+#### VM specifications
+* Instance Template: _TODO: Add GCP link when final resources are in place._
Review Comment:
Calling out this TODO since we should resolve before merging if possible, or create an issue if not
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function will return the number of online and offline runners for
+// each OS (Windows, linux), an additional Github actions workflow will run
+// to request this Cloud Function and send alerts based on the status.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+async function monitorRunnerStatus() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_NAME,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let runners = await octokit.request(`GET /orgs/${process.env.ORG}/actions/runners`, {
+ org: process.env.ORG,
+ per_page: 100, // In order to avoid cropped results we are explicitly setting this option in combination with a daily cleanup
Review Comment:
We never expect to hit close to 100 runners, right?
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function will return the number of online and offline runners for
+// each OS (Windows, linux), an additional Github actions workflow will run
+// to request this Cloud Function and send alerts based on the status.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+async function monitorRunnerStatus() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_NAME,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let runners = await octokit.request(`GET /orgs/${process.env.ORG}/actions/runners`, {
+ org: process.env.ORG,
+ per_page: 100, // In order to avoid cropped results we are explicitly setting this option in combination with a daily cleanup
+ });
+
+ //Filtering BEAM runners
+ let beamRunners = runners.data.runners.filter(runner => {
+ for (let label of runner.labels) {
+ if (label.name == "beam") {
+ return true;
+ }
+ }
+ return false;
Review Comment:
```suggestion
return runner.labels.find(label => label.name == "beam")
```
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function will return the number of online and offline runners for
+// each OS (Windows, linux), an additional Github actions workflow will run
+// to request this Cloud Function and send alerts based on the status.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+async function monitorRunnerStatus() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_NAME,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let runners = await octokit.request(`GET /orgs/${process.env.ORG}/actions/runners`, {
+ org: process.env.ORG,
+ per_page: 100, // In order to avoid cropped results we are explicitly setting this option in combination with a daily cleanup
+ });
+
+ //Filtering BEAM runners
+ let beamRunners = runners.data.runners.filter(runner => {
+ for (let label of runner.labels) {
+ if (label.name == "beam") {
+ return true;
+ }
+ }
+ return false;
Review Comment:
Is there any chance these won't be beam runners?
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function will return the number of online and offline runners for
+// each OS (Windows, linux), an additional Github actions workflow will run
+// to request this Cloud Function and send alerts based on the status.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+async function monitorRunnerStatus() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_NAME,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let runners = await octokit.request(`GET /orgs/${process.env.ORG}/actions/runners`, {
+ org: process.env.ORG,
+ per_page: 100, // In order to avoid cropped results we are explicitly setting this option in combination with a daily cleanup
+ });
+
+ //Filtering BEAM runners
+ let beamRunners = runners.data.runners.filter(runner => {
+ for (let label of runner.labels) {
+ if (label.name == "beam") {
+ return true;
+ }
+ }
+ return false;
+ });
+
+ //Dividing status for each runner OS
+ let osList = ["Linux", "Windows"];
+ let status = {}
+ for (let os of osList) {
+ let osRunners = beamRunners.filter(runner => {
+ for (let label of runner.labels) {
+ if (label.name == os) {
+ return true;
+ }
+ }
+ return false;
+ });
Review Comment:
```suggestion
let osRunners = beamRunners.filter(runner => {
return runner.labels.find(label => label.name == os)
});
```
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function will return the number of online and offline runners for
+// each OS (Windows, linux), an additional Github actions workflow will run
+// to request this Cloud Function and send alerts based on the status.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+async function monitorRunnerStatus() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_NAME,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let runners = await octokit.request(`GET /orgs/${process.env.ORG}/actions/runners`, {
+ org: process.env.ORG,
Review Comment:
Should we add some validation that `process.env.ORG` is set when this file is executed?
##########
.github/gh-actions-self-hosted-runners/self-hosted-linux/README.md:
##########
@@ -0,0 +1,100 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# GitHub Actions - Self-hosted Linux Runners
+These folders contain the required resources to deploy the GitHub Actions self-hosted runners for the workflows running in Ubuntu OS.
+* /docker
+ * Dockerfile and entrypoint: Resources needed to create a new self-hosted-runner image.
+ * docker-compose.yml: In case you would like to test and run the self-hosted runner locally.
+
+* /kubernetes
+ * Kubernetes files to create the resources needed to deploy the self-hosted runners.
+
+## Docker
+
+#### How to build a new image and push it to the Artifact Registry?
+* Create the image locally
+
+`docker build -t $LOCAL_IMAGE_NAME:TAG .`
+
+* Tag the local image with the GCP repository name
+
+`docker tag $LOCAL_IMAGE_NAME LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest`
+
+* Make sure you are authenticated to the gcloud repository
+
+`gcloud auth configure-docker us-central1-docker.pkg.dev`
+
+* Push the tagged image to the repository
+
+`docker push LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest`
+
+* _**Important: Please add the commit hash as a tag when a new image is uploaded.**_
+
+#### How to run a self-hosted locally?
Review Comment:
```suggestion
#### How to run a self-hosted runner locally?
```
##########
.github/gh-actions-self-hosted-runners/self-hosted-windows/shutdownScript.ps1:
##########
@@ -0,0 +1,24 @@
+# <!--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -->
+
+Write-Output "removingRunner"
+Set-Location C:/actionsDir
+
+$token=[System.Environment]::GetEnvironmentVariable('GITHUB_TOKEN','machine')
+Write-Output $token
Review Comment:
Why are we writing this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
elink21 commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r970143930
##########
.github/gh-actions-self-hosted-runners/README.md:
##########
@@ -0,0 +1,121 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+# GitHub Actions - Self-hosted Runners
+The current GitHub Actions workflows are being tested on multiple operating systems, such as Ubuntu, Windows and MacOS. The way to migrate these runners from GitHub to GCP is by implementing self-hosted runners, so we implemented them in both Ubuntu and Windows environments, going with Google Kubernetes Engine and Google Cloud Compute VMs instances respectively.
+
+On the other hand, we will rely on GitHub-hosted runners for MacOS builds until a straightforward implementation approach comes out.
+
+## Ubuntu
+Ubuntu Self-hosted runners are stored in Artifact Registry and implemented using Google Kubernetes Engine with the following specifications:
+
+#### Cluster
+* Cluster: [gh-actions-linux-runners](https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/gh-actions-linux-runners/details?project=apache-beam-testing)
+* Image: [linux-github-actions-runner](https://console.cloud.google.com/artifacts/docker/apache-beam-testing/us-central1/beam-github-actions/linux-github-actions-runner?project=apache-beam-testing)
+
+#### Pool
+* Number of nodes: 5
+* Cluster Autoscaler: ON
+ * Minimum number of nodes: 5
+ * Maximum number of nodes: 10
+
+#### Node
+* Machine Type: e2-custom-6-18432
+* Disk Size: 100 GB
+* CPU: 6 vCPUs
+* Memory : 18 GB
+
+#### Pod
+##### Container 1: gh-actions-runner
+* Image: $LOCAL_IMAGE_NAME LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest
+* CPU: 2
+* Memory: 1028 Mi
+* Volumes
+ * gcloud-key
+ * docker-certs-client
+* Environment variables
+ * Container variables
+ * GOOGLE_APPLICATION_CREDENTIALS
+ * DOCKER_HOST
+ * DOCKER_TLS_VERIFY
+ * DOCKER_CERT_PATH
+ * Kubernetes secret env variables
+ * github-actions-secrets
+ * gcloud-key
+
+##### Container 2: dind
+* Image: `docker:20.10.17-dind`
+* Volumes
+ * dind-storage
+ * docker-certs-client
+
+###### Pod Diagram
+
+
+#### AutoScaling
+* Horizontal Pod Autoscaling
+ * 5-10 nodes (From Pool Cluster Autoscaler)
+ * HorizontalPodAutoscaler
+ * Min replicas: 10
+ * Max replicas: 20
+ * CPU utilization: 70%
+* Vertical Pod Autoscaling
+ * updateMode: "Auto"
+
+
+## Windows
+Windows Virtual machines have the following specifications
+
+#### VM specifications
+* Instance Template: _TODO: Add GCP link when final resources are in place._
Review Comment:
We are adding the template name instead since a hardcoded link will become outdated quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
elink21 commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r970095051
##########
.github/gh-actions-self-hosted-runners/self-hosted-windows/shutdownScript.ps1:
##########
@@ -0,0 +1,24 @@
+# <!--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+
+# http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# -->
+
+Write-Output "removingRunner"
+Set-Location C:/actionsDir
+
+$token=[System.Environment]::GetEnvironmentVariable('GITHUB_TOKEN','machine')
+Write-Output $token
Review Comment:
Will be removed, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r971315786
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/index.js:
##########
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function provides registration tokens for windows and linux self-hosted
+//runners, a service account with the appropriated permissions should be used to
+// invoke the Cloud Function.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+const REQUIRED_ENV_VARS=["APP_ID","PEM_KEY","CLIENT_ID","CLIENT_SECRET","APP_INSTALLATION_ID","ORG"]
+
+
Review Comment:
Removed blank spaces from the `js` scripts, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r971298889
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/index.js:
##########
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function provides registration tokens for windows and linux self-hosted
+//runners, a service account with the appropriated permissions should be used to
+// invoke the Cloud Function.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+const REQUIRED_ENV_VARS=["APP_ID","PEM_KEY","CLIENT_ID","CLIENT_SECRET","APP_INSTALLATION_ID","ORG"]
Review Comment:
Yes, it's the same. We are using these `js` scripts in Cloud Functions so we did a couple of changes in here, like removing the `http` function, in order to give visibility to the community and to make the functions work in their locals.
We can add `REQUIRED_ENV_VARS` as an import here in the code but I think that in the Cloud Functions the change might not be needed as we will need an extra GCP service to share resources/data, likewise having an extra `js` file for one constant will increase the redundancy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23158:
URL: https://github.com/apache/beam/pull/23158#issuecomment-1242466080
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @Abacn for label build.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
elink21 commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r970145627
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function will return the number of online and offline runners for
+// each OS (Windows, linux), an additional Github actions workflow will run
+// to request this Cloud Function and send alerts based on the status.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+async function monitorRunnerStatus() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_NAME,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let runners = await octokit.request(`GET /orgs/${process.env.ORG}/actions/runners`, {
+ org: process.env.ORG,
+ per_page: 100, // In order to avoid cropped results we are explicitly setting this option in combination with a daily cleanup
+ });
+
+ //Filtering BEAM runners
+ let beamRunners = runners.data.runners.filter(runner => {
+ for (let label of runner.labels) {
+ if (label.name == "beam") {
+ return true;
+ }
+ }
+ return false;
Review Comment:
Yes, I'm going to change the code to navigate through all existent pages , thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
elink21 commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r970093131
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function will return the number of online and offline runners for
+// each OS (Windows, linux), an additional Github actions workflow will run
+// to request this Cloud Function and send alerts based on the status.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+async function monitorRunnerStatus() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_NAME,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let runners = await octokit.request(`GET /orgs/${process.env.ORG}/actions/runners`, {
+ org: process.env.ORG,
+ per_page: 100, // In order to avoid cropped results we are explicitly setting this option in combination with a daily cleanup
Review Comment:
Yes, it's possible cause we are sharing org with other projects and unfortunately we can only filter after the request is made, I'm going to change the code to navigate through all existent pages , thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r971316494
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/index.js:
##########
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function provides registration tokens for windows and linux self-hosted
+//runners, a service account with the appropriated permissions should be used to
+// invoke the Cloud Function.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+const REQUIRED_ENV_VARS=["APP_ID","PEM_KEY","CLIENT_ID","CLIENT_SECRET","APP_INSTALLATION_ID","ORG"]
Review Comment:
Added `REQUIRED_ENV_VARS` as a shared constant, could you review the code again please?
Note: This won't be added in the Cloud Functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm merged pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
damccorm merged PR #23158:
URL: https://github.com/apache/beam/pull/23158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
elink21 commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r970095819
##########
.github/gh-actions-self-hosted-runners/README.md:
##########
@@ -0,0 +1,121 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+# GitHub Actions - Self-hosted Runners
+The current GitHub Actions workflows are being tested on multiple operating systems, such as Ubuntu, Windows and MacOS. The way to migrate these runners from GitHub to GCP is by implementing self-hosted runners, so we implemented them in both Ubuntu and Windows environments, going with Google Kubernetes Engine and Google Cloud Compute VMs instances respectively.
+
+On the other hand, we will rely on GitHub-hosted runners for MacOS builds until a straightforward implementation approach comes out.
+
+## Ubuntu
+Ubuntu Self-hosted runners are stored in Artifact Registry and implemented using Google Kubernetes Engine with the following specifications:
+
+#### Cluster
+* Cluster: [gh-actions-linux-runners](https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/gh-actions-linux-runners/details?project=apache-beam-testing)
+* Image: [linux-github-actions-runner](https://console.cloud.google.com/artifacts/docker/apache-beam-testing/us-central1/beam-github-actions/linux-github-actions-runner?project=apache-beam-testing)
+
+#### Pool
+* Number of nodes: 5
+* Cluster Autoscaler: ON
+ * Minimum number of nodes: 5
+ * Maximum number of nodes: 10
+
+#### Node
+* Machine Type: e2-custom-6-18432
+* Disk Size: 100 GB
+* CPU: 6 vCPUs
+* Memory : 18 GB
+
+#### Pod
+##### Container 1: gh-actions-runner
+* Image: $LOCAL_IMAGE_NAME LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:latest
+* CPU: 2
+* Memory: 1028 Mi
+* Volumes
+ * gcloud-key
+ * docker-certs-client
+* Environment variables
+ * Container variables
+ * GOOGLE_APPLICATION_CREDENTIALS
+ * DOCKER_HOST
+ * DOCKER_TLS_VERIFY
+ * DOCKER_CERT_PATH
+ * Kubernetes secret env variables
+ * github-actions-secrets
+ * gcloud-key
+
+##### Container 2: dind
+* Image: `docker:20.10.17-dind`
+* Volumes
+ * dind-storage
+ * docker-certs-client
+
+###### Pod Diagram
+
+
+#### AutoScaling
+* Horizontal Pod Autoscaling
+ * 5-10 nodes (From Pool Cluster Autoscaler)
+ * HorizontalPodAutoscaler
+ * Min replicas: 10
+ * Max replicas: 20
+ * CPU utilization: 70%
+* Vertical Pod Autoscaling
+ * updateMode: "Auto"
+
+
+## Windows
+Windows Virtual machines have the following specifications
+
+#### VM specifications
+* Instance Template: _TODO: Add GCP link when final resources are in place._
Review Comment:
We have resolved the image nomenclature, it's going to be added, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r971248793
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/index.js:
##########
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function provides registration tokens for windows and linux self-hosted
+//runners, a service account with the appropriated permissions should be used to
+// invoke the Cloud Function.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+const REQUIRED_ENV_VARS=["APP_ID","PEM_KEY","CLIENT_ID","CLIENT_SECRET","APP_INSTALLATION_ID","ORG"]
+
+
Review Comment:
```suggestion
const REQUIRED_ENV_VARS=["APP_ID","PEM_KEY","CLIENT_ID","CLIENT_SECRET","APP_INSTALLATION_ID","ORG"]
```
Nit: remove extra space here and elsewhere
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js:
##########
@@ -0,0 +1,98 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function will return the number of online and offline runners for
+// each OS (Windows, linux), an additional Github actions workflow will run
+// to request this Cloud Function and send alerts based on the status.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+
+const REQUIRED_ENV_VARS=["APP_ID","PEM_KEY","CLIENT_ID","CLIENT_SECRET","APP_INSTALLATION_ID","ORG"]
+
+
+function validateEnvSet(envVars) {
+ envVars.forEach(envVar => {
+ if (!process.env[envVar]) {
+ throw new Error(`${envVar} environment variable not set.`)
+ }
+ });
+}
+
+async function monitorRunnerStatus() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_SECRET,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let pageCounter=1
+ let runners=[]
+ let pageRunners=[]
+ do{
+ pageRunners= await octokit.request(`GET /orgs/${process.env.ORG}/actions/runners`, {
+ org: process.env.ORG,
+ per_page: 50,
+ page:pageCounter
+ });
+ runners=runners.concat(pageRunners.data.runners)
+ pageCounter++
+ } while(pageRunners.data.runners.length!=0)
+
Review Comment:
```suggestion
const runners = await octokit.paginate("GET /orgs/${process.env.ORG}/actions/runners", {
org: process.env.ORG,
})
```
Could we simplify this with octokit's built in pagination?
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/generateToken/index.js:
##########
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function provides registration tokens for windows and linux self-hosted
+//runners, a service account with the appropriated permissions should be used to
+// invoke the Cloud Function.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+const REQUIRED_ENV_VARS=["APP_ID","PEM_KEY","CLIENT_ID","CLIENT_SECRET","APP_INSTALLATION_ID","ORG"]
Review Comment:
This is the same env set across files, right? Could we refactor it to a constant that we can just import?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r970057649
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function will return the number of online and offline runners for
+// each OS (Windows, linux), an additional Github actions workflow will run
+// to request this Cloud Function and send alerts based on the status.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+async function monitorRunnerStatus() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_NAME,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let runners = await octokit.request(`GET /orgs/${process.env.ORG}/actions/runners`, {
+ org: process.env.ORG,
+ per_page: 100, // In order to avoid cropped results we are explicitly setting this option in combination with a daily cleanup
+ });
+
+ //Filtering BEAM runners
+ let beamRunners = runners.data.runners.filter(runner => {
+ for (let label of runner.labels) {
+ if (label.name == "beam") {
+ return true;
+ }
+ }
+ return false;
Review Comment:
Ah, in that case should we use `octokit.paginate` to make sure we're getting all the results (not just the first 100)? https://octokit.github.io/rest.js/v18#pagination Potentially there could eventually be lots of them, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
elink21 commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r971028233
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function will return the number of online and offline runners for
+// each OS (Windows, linux), an additional Github actions workflow will run
+// to request this Cloud Function and send alerts based on the status.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+async function monitorRunnerStatus() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_NAME,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let runners = await octokit.request(`GET /orgs/${process.env.ORG}/actions/runners`, {
+ org: process.env.ORG,
Review Comment:
Yes, we have now included an env validation block, could you help us with an additional review for this and the other modifications? Thanks in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on PR #23158:
URL: https://github.com/apache/beam/pull/23158#issuecomment-1245726995
Hi @Abacn @damccorm, could you help us with a PR review please?
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dannymartinm commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
dannymartinm commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r970045015
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function will return the number of online and offline runners for
+// each OS (Windows, linux), an additional Github actions workflow will run
+// to request this Cloud Function and send alerts based on the status.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+async function monitorRunnerStatus() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_NAME,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let runners = await octokit.request(`GET /orgs/${process.env.ORG}/actions/runners`, {
+ org: process.env.ORG,
+ per_page: 100, // In order to avoid cropped results we are explicitly setting this option in combination with a daily cleanup
+ });
+
+ //Filtering BEAM runners
+ let beamRunners = runners.data.runners.filter(runner => {
+ for (let label of runner.labels) {
+ if (label.name == "beam") {
+ return true;
+ }
+ }
+ return false;
Review Comment:
Yes, as we are registering the self-hosted runners at an Organization level in a Runner Group. When calling to the [List self-hosted runners for an organization](https://docs.github.com/en/rest/actions/self-hosted-runners#list-self-hosted-runners-for-an-organization) GET function to retrieve the registered self-hosted runners, there is no way to filter by Runner Group. So we get all the self-hosted runners and here in this function we are getting the runners that has the label `beam`.
We are adding that tag in the `entrypoint.sh` when adding the `ubuntu-20.04` tag too.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on a diff in pull request #23158: [GitHub Actions] - INFRA scripts to implement GCP Self-hosted runners (Ubuntu&Windows)
Posted by GitBox <gi...@apache.org>.
elink21 commented on code in PR #23158:
URL: https://github.com/apache/beam/pull/23158#discussion_r970093131
##########
.github/gh-actions-self-hosted-runners/helper-functions/cloud-functions/monitorRunnersStatus/index.js:
##########
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//This function will return the number of online and offline runners for
+// each OS (Windows, linux), an additional Github actions workflow will run
+// to request this Cloud Function and send alerts based on the status.
+
+import functions from '@google-cloud/functions-framework';
+import { Octokit } from "octokit";
+import { createAppAuth } from "@octokit/auth-app";
+
+async function monitorRunnerStatus() {
+ try {
+ //Set your GH App values as environment variables
+ let authOptions = {
+ appId: process.env.APP_ID,
+ privateKey: process.env.PEM_KEY,
+ clientId: process.env.CLIENT_ID,
+ clientSecret: process.env.CLIENT_NAME,
+ installationId: process.env.APP_INSTALLATION_ID
+ }
+ const octokit = new Octokit({
+ authStrategy: createAppAuth,
+ auth: authOptions
+ });
+ let runners = await octokit.request(`GET /orgs/${process.env.ORG}/actions/runners`, {
+ org: process.env.ORG,
+ per_page: 100, // In order to avoid cropped results we are explicitly setting this option in combination with a daily cleanup
Review Comment:
Yes, it's possible because we are sharing org with other projects and unfortunately we can only filter after the request is made, I'm going to change the code to navigate through all existent pages , thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org