You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/13 14:36:43 UTC
[GitHub] [beam] MakarkinSAkvelon opened a new pull request, #23208: [Playground] Helm+kubernetes+deployment
MakarkinSAkvelon opened a new pull request, #23208:
URL: https://github.com/apache/beam/pull/23208
New solution for Playground backend - with Kubernetes and Helm
New pipeline for Backend deployment: using HELM and Kubernetes
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989097806
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
+* Service account with the following roles should be created:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
+* Following APIs should be enabled:
+ - Identity and Access Management (IAM)
+ - Compute Engine API
+ - App Engine Admin API
+ - Cloud Resource Manager API
+
+* Make necessary changes in `playground/terraform/environment/beta/terraform.tfvars` file:
```
-
-## 1. Create new environment
-
-To provide information about the terraform backend, run the following commands
-
-* New environment folder
-
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+network_name = "network_name" #Choose network name
+project_id = "project_id" #Input project ID
+gke_name = "playground-backend" #Define GKE name
+region = "us-east1" #Choose region
+pg_location = "us-east1-b" #Choose location (should be in the region)
+state_bucket = "bucket_name" #Input bucket name
+bucket_examples_name = "bucket_name-example" #Input example bucket name
```
-
-* Backend config
-
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
+* Make necessary changes in `playground/terraform/environment/beta/state.tfbackend` file:
```
-
-* Terraform variables config and provide necessary variables
-
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
+bucket = "bucket_name" #input bucket name (will be used for tfstate file)
```
-
-Then provide necessary variables.
-
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
+* Export GOOGLE_APPLICATION_CREDENTIALS using following command:
```
-
-## 3. Deploy application
-
-To deploy application run following steps:
-
-* Authinticate in Artifact registry
-
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+ export GOOGLE_APPLICATION_CREDENTIALS=`your json key locaton`
```
-
-* Вeploy backend services
-
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+* Activate created service account using following command:
+```
+ gcloud auth activate-service-account `full principal service account` --key-file=`your json key locaton`
+```
+* Install kubectl:
+```
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+```
+* Install Helm:
+```
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+```
+# Infrastructure deployment:
+* Run following command for infrastructure deployment (please be sure that you are in the "beam" folder):
+```
+./gradlew playground:terraform:InitInfrastructure -Pproject_environment="beta"
+```
+# Backend deployment:
+* Login to Docker registry:
+```
+ cat `your json key locaton` | docker login -u _json_key --password-stdin https://`chosen_region`-docker.pkg.dev
Review Comment:
https://cloud.google.com/container-registry/docs/advanced-authentication (authentication using JSON file)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989105951
##########
playground/infrastructure/helm-backend/values.yaml:
##########
@@ -0,0 +1,36 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+replicaCount: 1
+image:
+ java_image: beam_playground-backend-java
+ go_image: beam_playground-backend-go
+ router_image: beam_playground-backend-router
+ scio_image: beam_playground-backend-scio
+ python_image: beam_playground-backend-python
+ pullPolicy: Always
+ tag: "beta"
+
+service:
+ type: NodePort
+ targetPort: 8080
+ port: 443
+
+autoscaling:
+ enabled: false
Review Comment:
we're not ready yet (with scaling), moreover, we have only one GKE node for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
olehborysevych commented on PR #23208:
URL: https://github.com/apache/beam/pull/23208#issuecomment-1252208047
lgtm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r991886682
##########
playground/terraform/infrastructure/appengine/main.tf:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+resource "google_app_engine_application" "app_playground" {
+ project = var.project_id
+ location_id = var.region == "us-central1" ? var.location_id_us : var.region
Review Comment:
There is a problem with locations: for europe-west1 and us-central1 location name for AppEngine should be europe-west and us-central. So, this option (line 18) just helps us to identify and put correct location for AppEngine location:
https://cloud.google.com/appengine/docs/standard/locations#:~:text=If%20you%20already%20created%20an,the%20top%20of%20the%20page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
olehborysevych commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r991066421
##########
.github/workflows/playground_deploy_backend_gke.yml:
##########
@@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: kuber-bkend
+on:
+ workflow_dispatch:
+
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ DOCKERTAG: beta
+jobs:
+ deployBackend:
+ name: init
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ - name: Install kubectl
+ run: |
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+ - name: Install helm
+ run: |
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+ - name: Set up Cloud SDK
+ uses: google-github-actions/setup-gcloud@v0
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.S3D_GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
+ which gcloud
+ gcloud auth activate-service-account --project=${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }} --key-file=/tmp/gcp_access.json
+ - name: Update variables
+ run: |
+ echo "redis_ip: $(gcloud redis instances list --region us-central1 --format="value(host)"):6379" >> playground/infrastructure/helm-backend/values.yaml
+ echo "project_id: "${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "registry: "${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip: $(gcloud compute addresses list --filter "NAME:${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static" --format="value(address)")" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip_name: ${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static" >> playground/infrastructure/helm-backend/values.yaml
+ cat playground/infrastructure/helm-backend/values.yaml
+ - name: Get K8s Config
+ run: gcloud container clusters get-credentials --region us-central1-a playground-examples
+ - name: Login to Docker Registry
+ run: |
+ cat /tmp/gcp_access.json | docker login -u _json_key --password-stdin https://us-central1-docker.pkg.dev
+ - name: Build And Push Backend
+ run: ./gradlew playground:backend:containers:java:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pbase-image='apache/beam_java8_sdk:${{ env.BEAM_VERSION }}' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Go Backend
+ run: ./gradlew playground:backend:containers:go:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Python Backend
+ run: ./gradlew playground:backend:containers:python:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Scio Backend
+ run: ./gradlew playground:backend:containers:scio:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Router Backend
+ run: ./gradlew playground:backend:containers:router:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Index creation
+ run: |
+ gcloud app deploy playground/index.yaml --project=${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}
Review Comment:
@MakarkinSAkvelon this still uses your secret S3D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
olehborysevych commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r974960087
##########
.github/workflows/k8e-backend.yml:
##########
@@ -0,0 +1,85 @@
+#
Review Comment:
@MakarkinSAkvelon I think we need to rename this file to something like playground_deploy_backend
##########
.github/workflows/k8e-backend.yml:
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: build_playground_backend_kubernetes
+on:
+ push:
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ K8S_NAMESPACE: playground-backend
+ HELM_APP_NAME: playground-backend
+ DOCKERTAG: beta
+jobs:
+ deployBackend:
+ name: Start to Deploy
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ - name: Install kubectl
+ run: |
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+ - name: Install helm
+ run: |
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+ - name: Set up Cloud SDK
+ uses: google-github-actions/setup-gcloud@v0
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.S3D_GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
+ which gcloud
+ gcloud auth activate-service-account --project=${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }} --key-file=/tmp/gcp_access.json
+ - name: create static ip
+ run: gcloud compute addresses create ${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static --global
+ - name: Update variables
+ run: |
+ echo "redis_ip: $(gcloud redis instances list --region us-central1 --format="value(host)"):6379" >> playground/infrastructure/helm-backend/values.yaml
+ echo "project_id: "${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "registry: "${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip: $(gcloud compute addresses list --filter "NAME:${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static" --format="value(address)")" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip_name: ${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static" >> playground/infrastructure/helm-backend/values.yaml
+ cat playground/infrastructure/helm-backend/values.yaml
+ - name: Get K8s Config
+ run: gcloud container clusters get-credentials --region us-central1-a playground-examples
+ - name: Login to Docker Registry
+ run: |
+ cat /tmp/gcp_access.json | docker login -u _json_key --password-stdin https://us-central1-docker.pkg.dev
Review Comment:
docker registry is hardcode
##########
playground/infrastructure/helm-backend/templates/go.yml:
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: playground-go
Review Comment:
I would like to see more meaningfull filenames here. "deployment-go" instead of "go" and the same for the rest of files
##########
playground/infrastructure/helm-backend/Chart.yaml:
##########
@@ -0,0 +1,22 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v2
+description: A Helm chart for Apache Beam Application Deployment
Review Comment:
name is misleading
##########
.github/workflows/k8e-backend.yml:
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: build_playground_backend_kubernetes
+on:
+ push:
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ K8S_NAMESPACE: playground-backend
+ HELM_APP_NAME: playground-backend
+ DOCKERTAG: beta
+jobs:
+ deployBackend:
+ name: Start to Deploy
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ - name: Install kubectl
+ run: |
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+ - name: Install helm
+ run: |
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+ - name: Set up Cloud SDK
+ uses: google-github-actions/setup-gcloud@v0
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.S3D_GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
+ which gcloud
+ gcloud auth activate-service-account --project=${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }} --key-file=/tmp/gcp_access.json
+ - name: create static ip
+ run: gcloud compute addresses create ${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static --global
+ - name: Update variables
+ run: |
+ echo "redis_ip: $(gcloud redis instances list --region us-central1 --format="value(host)"):6379" >> playground/infrastructure/helm-backend/values.yaml
+ echo "project_id: "${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "registry: "${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip: $(gcloud compute addresses list --filter "NAME:${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static" --format="value(address)")" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip_name: ${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static" >> playground/infrastructure/helm-backend/values.yaml
+ cat playground/infrastructure/helm-backend/values.yaml
+ - name: Get K8s Config
+ run: gcloud container clusters get-credentials --region us-central1-a playground-examples
Review Comment:
region name and "playground-examples" is hardcoded here and I think the name is misleading
##########
.github/workflows/k8e-backend.yml:
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: build_playground_backend_kubernetes
+on:
+ push:
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ K8S_NAMESPACE: playground-backend
+ HELM_APP_NAME: playground-backend
+ DOCKERTAG: beta
+jobs:
+ deployBackend:
+ name: Start to Deploy
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ - name: Install kubectl
+ run: |
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+ - name: Install helm
+ run: |
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+ - name: Set up Cloud SDK
+ uses: google-github-actions/setup-gcloud@v0
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.S3D_GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
+ which gcloud
+ gcloud auth activate-service-account --project=${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }} --key-file=/tmp/gcp_access.json
+ - name: create static ip
+ run: gcloud compute addresses create ${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static --global
+ - name: Update variables
+ run: |
+ echo "redis_ip: $(gcloud redis instances list --region us-central1 --format="value(host)"):6379" >> playground/infrastructure/helm-backend/values.yaml
Review Comment:
this could be done with something like yaml-update-action from github
##########
.github/workflows/k8e-backend.yml:
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: build_playground_backend_kubernetes
+on:
+ push:
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ K8S_NAMESPACE: playground-backend
+ HELM_APP_NAME: playground-backend
+ DOCKERTAG: beta
+jobs:
+ deployBackend:
+ name: Start to Deploy
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ - name: Install kubectl
+ run: |
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+ - name: Install helm
+ run: |
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+ - name: Set up Cloud SDK
+ uses: google-github-actions/setup-gcloud@v0
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.S3D_GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
+ which gcloud
+ gcloud auth activate-service-account --project=${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }} --key-file=/tmp/gcp_access.json
+ - name: create static ip
+ run: gcloud compute addresses create ${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static --global
+ - name: Update variables
+ run: |
+ echo "redis_ip: $(gcloud redis instances list --region us-central1 --format="value(host)"):6379" >> playground/infrastructure/helm-backend/values.yaml
+ echo "project_id: "${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "registry: "${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip: $(gcloud compute addresses list --filter "NAME:${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static" --format="value(address)")" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip_name: ${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static" >> playground/infrastructure/helm-backend/values.yaml
+ cat playground/infrastructure/helm-backend/values.yaml
+ - name: Get K8s Config
+ run: gcloud container clusters get-credentials --region us-central1-a playground-examples
+ - name: Login to Docker Registry
+ run: |
+ cat /tmp/gcp_access.json | docker login -u _json_key --password-stdin https://us-central1-docker.pkg.dev
+ - name: Build And Push Backend
+ run: ./gradlew playground:backend:containers:java:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pbase-image='apache/beam_java8_sdk:${{ env.BEAM_VERSION }}' -Pdocker-tag=${{ env.DOCKERTAG }}
Review Comment:
playground-repository needs to be externalized to variable
##########
.github/workflows/k8e-backend.yml:
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: build_playground_backend_kubernetes
+on:
+ push:
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ K8S_NAMESPACE: playground-backend
+ HELM_APP_NAME: playground-backend
+ DOCKERTAG: beta
+jobs:
+ deployBackend:
+ name: Start to Deploy
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ - name: Install kubectl
+ run: |
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+ - name: Install helm
+ run: |
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+ - name: Set up Cloud SDK
+ uses: google-github-actions/setup-gcloud@v0
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.S3D_GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
Review Comment:
still has old secret and project id's key here and below
##########
playground/infrastructure/helm-backend/templates/go.yml:
##########
@@ -0,0 +1,45 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: playground-go
Review Comment:
ALso we discussed the MAX_JOBS environment variable
##########
playground/infrastructure/helm-backend/values.yaml:
##########
@@ -0,0 +1,42 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+replicaCount: 1
+image:
+ java_image: beam_playground-backend-java
+ go_image: beam_playground-backend-go
+ router_image: beam_playground-backend-router
+ scio_image: beam_playground-backend-scio
+ python_image: beam_playground-backend-python
+ pullPolicy: Always
+ tag: "beta"
+
+service:
+ type: NodePort
+ targetPort: 8080
+ port: 443
+
+autoscaling:
+ enabled: false
+ minReplicas: 1
+ maxReplicas: 4
+ targetCPUUtilizationPercentage: 80
+Footer
Review Comment:
this looks odd!
##########
playground/infrastructure/helm-backend/templates/service-python.yml:
##########
@@ -0,0 +1,40 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: Service
+metadata:
+ labels:
+ app: backend-python
+ type: backend
+ name: playground-python
+ annotations:
+ cloud.google.com/backend-config: '{"default": "bkend-health"}'
+ cloud.google.com/neg: '{"ingress": true}'
+spec:
+ ports:
+ - port: {{ .Values.service.port }}
+ protocol: TCP
+ targetPort: {{ .Values.service.targetPort }}
+ selector:
+ app: backend-python
+ type: {{ .Values.service.type }}
+Footer
Review Comment:
This looks odd
##########
.github/workflows/k8e-backend.yml:
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: build_playground_backend_kubernetes
+on:
+ push:
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ K8S_NAMESPACE: playground-backend
+ HELM_APP_NAME: playground-backend
+ DOCKERTAG: beta
+jobs:
+ deployBackend:
+ name: Start to Deploy
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ - name: Install kubectl
+ run: |
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+ - name: Install helm
+ run: |
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+ - name: Set up Cloud SDK
+ uses: google-github-actions/setup-gcloud@v0
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.S3D_GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
+ which gcloud
+ gcloud auth activate-service-account --project=${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }} --key-file=/tmp/gcp_access.json
+ - name: create static ip
+ run: gcloud compute addresses create ${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static --global
+ - name: Update variables
+ run: |
+ echo "redis_ip: $(gcloud redis instances list --region us-central1 --format="value(host)"):6379" >> playground/infrastructure/helm-backend/values.yaml
+ echo "project_id: "${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "registry: "${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip: $(gcloud compute addresses list --filter "NAME:${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static" --format="value(address)")" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip_name: ${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static" >> playground/infrastructure/helm-backend/values.yaml
+ cat playground/infrastructure/helm-backend/values.yaml
+ - name: Get K8s Config
+ run: gcloud container clusters get-credentials --region us-central1-a playground-examples
+ - name: Login to Docker Registry
+ run: |
+ cat /tmp/gcp_access.json | docker login -u _json_key --password-stdin https://us-central1-docker.pkg.dev
+ - name: Build And Push Backend
+ run: ./gradlew playground:backend:containers:java:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pbase-image='apache/beam_java8_sdk:${{ env.BEAM_VERSION }}' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Go Backend
+ run: ./gradlew playground:backend:containers:go:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Python Backend
+ run: ./gradlew playground:backend:containers:python:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Scio Backend
+ run: ./gradlew playground:backend:containers:scio:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Router Backend
+ run: ./gradlew playground:backend:containers:router:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Database and index creation
+ run: |
+ gcloud app create --region=us-central
Review Comment:
region is hardcoded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989102956
##########
playground/terraform/environment/beta/state.tfbackend:
##########
@@ -17,4 +17,4 @@
# under the License.
#
-bucket = "beam_playground_terraform"
+bucket = "pg-second"
Review Comment:
any name, doesn't matter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AKosolapov commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
AKosolapov commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r990211885
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisite:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install GCloud to your Operation system
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+When you create a new project, you must add a service account with a JSON key and a bucket according to the instructions above. Also, according to the instructions, configure the roles for the service account:
Review Comment:
Consider rephrasing:
After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select `state_bucket` for storing the Terraform state data according to the instructions above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon closed pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon closed pull request #23208: [Playground] Helm+kubernetes+deployment
URL: https://github.com/apache/beam/pull/23208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r994387273
##########
playground/terraform/infrastructure/appengine/main.tf:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+resource "google_app_engine_application" "app_playground" {
+ project = var.project_id
+ location_id = var.region == "us-central1" ? var.location_id_us : var.region
+ database_type = "CLOUD_DATASTORE_COMPATIBILITY"
+}
+
+resource "google_project_service" "firestore" {
+ project = var.project_id
+ service = "firestore.googleapis.com"
+ disable_dependent_services = true
+ depends_on = [
+ google_app_engine_application.app_playground
+ ]
+}
+
+resource "google_app_engine_flexible_app_version" "default_app" {
+ depends_on = [
+ google_app_engine_application.app_playground
+ ]
+ count = var.create_default_service ? 1 : 0
+ service = "default"
+ version_id = "mlflow-default"
+ runtime = "custom"
+ project = var.project_id
+
+ deployment {
+ container {
+ image = "gcr.io/cloudrun/hello"
Review Comment:
Was forced to return back this option because of the following error during Frontend deployment:
Error: Error creating FlexibleAppVersion: googleapi: Error 400: The first service (module) you upload to a new application must be the 'default' service (module). Please upload a version of the 'default' service (module) before uploading a version for the 'frontend-stg' service (module). See the documentation for more information. Python: (https://developers.google.com/appengine/docs/python/modules/#Python_Uploading%%20modules) Java: (https://developers.google.com/appengine/docs/java/modules/#Java_Uploading%%20modules)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r994390495
##########
.github/workflows/playground_deploy_infrastructure.yml:
##########
@@ -0,0 +1,18 @@
+name: New environment creation
+on:
+ workflow_dispatch:
+jobs:
+ Infrastructure_deployment:
+ runs-on: ubuntu-latest
+ env:
+ PROJECT_ENV: beta
+ steps:
+ - name: git clone repo
+ uses: actions/checkout@v2
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.S3D_GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r994219442
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,114 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. [How to create a new project in Google Cloud Platform](https://cloud.google.com/resource-manager/docs/creating-managing-projects/) _(Note: How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects)_
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. [How to create a new service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) _(Note: You will find the instruction "How to create Service account" for your new project)_
-# Deployment steps
+3. [How to create a JSON key for a Google Service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) _(Note: In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication)_
-## 0. Create GCS bucket for state
+4. [How to create a new bucket](https://cloud.google.com/storage/docs/creating-buckets) _(Note: Short instruction for bucket creation. You will need at least one bucket for file keeping)_
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
-```
+5. [How to assign a new role for the service account](https://cloud.google.com/iam/docs/granting-changing-revoking-access) _(Note: It will explain how to assign required roles for your service account)_
-## 1. Create new environment
+6. [How to install gcloud CLI to your local environment](https://cloud.google.com/sdk/docs/install?hl=en) _(Note: GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)_
-To provide information about the terraform backend, run the following commands
+7. [What are Google Cloud Platform APIs](https://cloud.google.com/apis) _(Note: Short description of all Google Platform APIs)_
-* New environment folder
+8. [Google Cloud Platform naming policy](https://cloud.google.com/compute/docs/naming-resources) _(Note: Describes the naming convention for Compute Engine resources)_
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
-```
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-* Backend config
+***Operation System preparation steps:***
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-* Terraform variables config and provide necessary variables
+* [Java](https://adoptopenjdk.net/)
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+* [Kubernetes Command Line Interface](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
-Then provide necessary variables.
+* [HELM](https://helm.sh/docs/intro/install/)
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+* [Docker](https://docs.docker.com/engine/install/)
-## 3. Deploy application
+* [Terraform](https://www.terraform.io/downloads)
-To deploy application run following steps:
+After installations of the required packages will be completed, you will need to download ***BEAM Playground*** from the repository.
-* Authinticate in Artifact registry
+Once downloading is completed you will find the "beam" folder on your disk (you can execute the `ls -la` command for Linux, or the `dir` command for Windows). Open that folder
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+It is possible to configure BEAM to work with the different versions of the environment (like staging, production, etc.), you will need to create a special folder in `playground/terraform/environment` and put all configuration files inside:
+* File name - `terraform.tfvars`, items inside:
```
+network_name = "network_name" #Enter the network name - you can choose any name for the network according to the naming policy above
+project_id = "project_id" #Enter the project ID - ID of created project
+gke_name = "playground-backend" #Set the GKE name - you can choose any name for Google Kubernetes Engine according to the naming policy above
+region = "us-east1" #Set the region - preferred region according to your needs
+location = "us-east1-b" #Select the location - location should be in region you set before
+state_bucket = "bucket_name" #Name of bucket - Google Cloud bucket where BEAM Playground will put temp files, [terraform state file] (https://spacelift.io/blog/terraform-state)
+bucket_examples_name = "bucket_name-example" #Enter an example bucket name - bucket for some build-in examples for BEAM Playground
+```
+* File name - `state.tfbackend`, item inside:
+```
+bucket = "bucket_name" #input bucket name - will be used for terraform tfstate file
+```
+Then, let's configure authentication for the Google Cloud Platform:
-* Вeploy backend services
-
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+* The following command allows us to authenticate using JSON key file
+```
+ export GOOGLE_APPLICATION_CREDENTIALS=`your service account JSON key location` (absolute path)
+```
+* Using the following command, we will activate the newly created service account:
+```
+ gcloud auth activate-service-account `full principal service account` --key-file=`your service account JSON key location` (absolute path)
```
-* Deploy frontend service
+# Infrastructure deployment:
+* To deploy the Infrastructure, use the following command (please be sure that you are in the "beam" folder):
+```
+./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env" (env - folder name which you created for configuration files)
+```
+# Backend deployment:
+Once the script was executed successfully, you will need to authenticate on Docker and Google Kubernetes Engine
+The following command will authorize us in the Docker registry
+```
+ cat `your service account json key locaton` | docker login -u _json_key --password-stdin https://`chosen_region`-docker.pkg.dev
+```
+* The following command will authenticate us in GKE
+```
+gcloud container clusters get-credentials --region `chosen_pg_location` `gke_name` --project `project_id`
+```
+* We need to create database indexes for BEAM playground examples by the following command:
+```
+gcloud app deploy playground/index.yaml --project=`project_id`
+```
+That's all, the configuration of the environment has been completed. For deploying the backend part to the Google cloud Kubernetes engine, please execute the following command (Ensure you are in the "beam" folder):
+```
+./gradlew playground:terraform:gkebackend -Pproject_environment="env" -Pdocker-tag="tag" (env - folder name which you created for configuration files, tag - image tag for backend)
Review Comment:
Hello, Damon
I hope you're switching branches from the master to the current one. Because, yes, till we merge it to the master branch, the helm folder will not be there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
olehborysevych commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r975180685
##########
.github/workflows/k8e-backend.yml:
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: build_playground_backend_kubernetes
+on:
+ push:
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ K8S_NAMESPACE: playground-backend
+ HELM_APP_NAME: playground-backend
+ DOCKERTAG: beta
+jobs:
+ deployBackend:
+ name: Start to Deploy
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ - name: Install kubectl
+ run: |
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+ - name: Install helm
+ run: |
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+ - name: Set up Cloud SDK
+ uses: google-github-actions/setup-gcloud@v0
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.S3D_GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
+ which gcloud
+ gcloud auth activate-service-account --project=${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }} --key-file=/tmp/gcp_access.json
+ - name: create static ip
+ run: gcloud compute addresses create ${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static --global
+ - name: Update variables
+ run: |
+ echo "redis_ip: $(gcloud redis instances list --region us-central1 --format="value(host)"):6379" >> playground/infrastructure/helm-backend/values.yaml
Review Comment:
TBD. Not needed for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] eantyshev commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
eantyshev commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r975214232
##########
.github/workflows/build_playground_backend_kubernetes.yml:
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: build_playground_backend_kubernetes
+on:
+ workflow_dispatch:
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ K8S_NAMESPACE: playground-backend
+ HELM_APP_NAME: playground-backend
+ DOCKERTAG: beta
+jobs:
+ deployBackend:
+ name: Start to Deploy
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ - name: Install kubectl
+ run: |
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+ - name: Install helm
+ run: |
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+ - name: Set up Cloud SDK
+ uses: google-github-actions/setup-gcloud@v0
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
+ which gcloud
+ gcloud auth activate-service-account --project=${{ secrets.GCP_PLAYGROUND_PROJECT_ID }} --key-file=/tmp/gcp_access.json
+ - name: create static ip
+ run: gcloud compute addresses create ${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}-static --global
+ - name: Update variables
+ run: |
+ echo "redis_ip: $(gcloud redis instances list --region us-central1 --format="value(host)"):6379" >> playground/infrastructure/helm-backend/values.yaml
+ echo "project_id: "${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "registry: "${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip: $(gcloud compute addresses list --filter "NAME:${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}-static" --format="value(address)")" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip_name: ${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}-static" >> playground/infrastructure/helm-backend/values.yaml
+ cat playground/infrastructure/helm-backend/values.yaml
+ - name: Get K8s Config
+ run: gcloud container clusters get-credentials --region us-central1-a playground-examples
Review Comment:
Another hard-coded variable: "playground-examples" could be also extracted from terraform output
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989104972
##########
playground/infrastructure/helm-backend/values.yaml:
##########
@@ -0,0 +1,36 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+replicaCount: 1
+image:
+ java_image: beam_playground-backend-java
+ go_image: beam_playground-backend-go
+ router_image: beam_playground-backend-router
+ scio_image: beam_playground-backend-scio
+ python_image: beam_playground-backend-python
+ pullPolicy: Always
+ tag: "beta"
Review Comment:
it's a variables in HELM, I don't think that we need to describe how it's work, because they shouldn't be changed by hands
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r992098288
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
Review Comment:
Fixed.
Add a new module which enable required APIs for BEAM Playground
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r992126640
##########
playground/terraform/infrastructure/variables.tf:
##########
@@ -167,5 +165,8 @@ variable "gke_name" {
variable "gke_location" {
description = "Location of GKE cluster"
- default = "us-central1-a"
+}
+
+variable "pg_location" {
Review Comment:
Fixed
##########
playground/terraform/infrastructure/appengine/main.tf:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+resource "google_app_engine_application" "app_playground" {
+ project = var.project_id
+ location_id = var.region == "us-central1" ? var.location_id_us : var.region
+ database_type = "CLOUD_DATASTORE_COMPATIBILITY"
+}
+
+resource "google_project_service" "firestore" {
+ project = var.project_id
+ service = "firestore.googleapis.com"
+ disable_dependent_services = true
+ depends_on = [
+ google_app_engine_application.app_playground
+ ]
+}
+
+resource "google_app_engine_flexible_app_version" "default_app" {
+ depends_on = [
+ google_app_engine_application.app_playground
+ ]
+ count = var.create_default_service ? 1 : 0
+ service = "default"
+ version_id = "mlflow-default"
+ runtime = "custom"
+ project = var.project_id
+
+ deployment {
+ container {
+ image = "gcr.io/cloudrun/hello"
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r993689877
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
Review Comment:
This step is described in "readme"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AKosolapov commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
AKosolapov commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r993943425
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,114 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. [How to create a new project in Google Cloud Platform](https://cloud.google.com/resource-manager/docs/creating-managing-projects/) _(Note: How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects)_
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. [How to create a new service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) _(Note: You will find the instruction "How to create Service account" for your new project)_
-# Deployment steps
+3. [How to create a JSON key for a Google Service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) _(Note: In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication)_
-## 0. Create GCS bucket for state
+4. [How to create a new bucket](https://cloud.google.com/storage/docs/creating-buckets) _(Note: Short instruction for bucket creation. You will need at least one bucket for file keeping)_
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
-```
+5. [How to assign a new role for the service account](https://cloud.google.com/iam/docs/granting-changing-revoking-access) _(Note: It will explain how to assign required roles for your service account)_
-## 1. Create new environment
+6. [How to install gcloud CLI to your local environment](https://cloud.google.com/sdk/docs/install?hl=en) _(Note: GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)_
-To provide information about the terraform backend, run the following commands
+7. [What are Google Cloud Platform APIs](https://cloud.google.com/apis) _(Note: Short description of all Google Platform APIs)_
-* New environment folder
+8. [Google Cloud Platform naming policy](https://cloud.google.com/compute/docs/naming-resources) _(Note: Describes the naming convention for Compute Engine resources)_
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
-```
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-* Backend config
+***Operation System preparation steps:***
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-* Terraform variables config and provide necessary variables
+* [Java](https://adoptopenjdk.net/)
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+* [Kubernetes Command Line Interface](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
-Then provide necessary variables.
+* [HELM](https://helm.sh/docs/intro/install/)
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+* [Docker](https://docs.docker.com/engine/install/)
-## 3. Deploy application
+* [Terraform](https://www.terraform.io/downloads)
-To deploy application run following steps:
+After installations of the required packages will be completed, you will need to download ***BEAM Playground*** from the repository.
-* Authinticate in Artifact registry
+Once downloading is completed you will find the "beam" folder on your disk (you can execute the `ls -la` command for Linux, or the `dir` command for Windows). Open that folder
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+It is possible to configure BEAM to work with the different versions of the environment (like staging, production, etc.), you will need to create a special folder in `playground/terraform/environment` and put all configuration files inside:
+* File name - `terraform.tfvars`, items inside:
```
+network_name = "network_name" #Enter the network name - you can choose any name for the network according to the naming policy above
+project_id = "project_id" #Enter the project ID - ID of created project
+gke_name = "playground-backend" #Set the GKE name - you can choose any name for Google Kubernetes Engine according to the naming policy above
+region = "us-east1" #Set the region - preferred region according to your needs
+location = "us-east1-b" #Select the location - location should be in region you set before
+state_bucket = "bucket_name" #Name of bucket - Google Cloud bucket where BEAM Playground will put temp files, [terraform state file] (https://spacelift.io/blog/terraform-state)
+bucket_examples_name = "bucket_name-example" #Enter an example bucket name - bucket for some build-in examples for BEAM Playground
+```
+* File name - `state.tfbackend`, item inside:
+```
+bucket = "bucket_name" #input bucket name - will be used for terraform tfstate file
+```
+Then, let's configure authentication for the Google Cloud Platform:
-* Вeploy backend services
-
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+* The following command allows us to authenticate using JSON key file
+```
+ export GOOGLE_APPLICATION_CREDENTIALS=`your service account JSON key location` (absolute path)
+```
+* Using the following command, we will activate the newly created service account:
+```
+ gcloud auth activate-service-account `full principal service account` --key-file=`your service account JSON key location` (absolute path)
```
-* Deploy frontend service
+# Infrastructure deployment:
+* To deploy the Infrastructure, use the following command (please be sure that you are in the "beam" folder):
+```
+./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env" (env - folder name which you created for configuration files)
+```
+# Backend deployment:
+Once the script was executed successfully, you will need to authenticate on Docker and Google Kubernetes Engine
+The following command will authorize us in the Docker registry
+```
+ cat `your service account json key locaton` | docker login -u _json_key --password-stdin https://`chosen_region`-docker.pkg.dev
+```
+* The following command will authenticate us in GKE
+```
+gcloud container clusters get-credentials --region `chosen_pg_location` `gke_name` --project `project_id`
+```
+* We need to create database indexes for BEAM playground examples by the following command:
+```
+gcloud app deploy playground/index.yaml --project=`project_id`
+```
+That's all, the configuration of the environment has been completed. For deploying the backend part to the Google cloud Kubernetes engine, please execute the following command (Ensure you are in the "beam" folder):
+```
+./gradlew playground:terraform:gkebackend -Pproject_environment="env" -Pdocker-tag="tag" (env - folder name which you created for configuration files, tag - image tag for backend)
Review Comment:
@damondouglas not clear from error message snippet - was command run from beam folder (per note above)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damondouglas commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
damondouglas commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r991489563
##########
playground/terraform/build.gradle.kts:
##########
@@ -395,3 +372,111 @@ task("deployBackend") {
dependsOn(deploy)
}
+task("takeConfig") {
+ group = "deploy"
+ doLast {
+ var ipaddr = ""
+ var redis = ""
+ var proj = ""
+ var registry = ""
+ var ipaddrname = ""
+ var d_tag = ""
+ var stdout = ByteArrayOutputStream()
+ if (project.hasProperty("docker-tag")) {
+ d_tag = project.property("docker-tag") as String
+ }
+ exec {
+ commandLine = listOf("terraform", "output", "playground_static_ip_address")
+ standardOutput = stdout
+ }
+ ipaddr = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+
+ exec {
+ commandLine = listOf("terraform", "output", "playground_redis_ip")
+ standardOutput = stdout
+ }
+ redis = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "playground_gke_project")
+ standardOutput = stdout
+ }
+ proj = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "docker-repository-root")
+ standardOutput = stdout
+ }
+ registry = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "playground_static_ip_address_name")
+ standardOutput = stdout
+ }
+ ipaddrname = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+
+ val configFileName = "values.yaml"
+ val modulePath = project(":playground").projectDir.absolutePath
+ val file = File("$modulePath/infrastructure/helm-backend/$configFileName")
+ val lines = file.readLines()
+ val endOfSlice = lines.indexOfFirst { it.contains("static_ip") }
+ if (endOfSlice != -1) {
+ val oldContent = lines.slice(0 until endOfSlice)
+ val flagDelete = file.delete()
+ if (!flagDelete) {
+ throw kotlin.RuntimeException("Deleting file failed")
+ }
+ val sb = kotlin.text.StringBuilder()
+ val lastLine = oldContent[oldContent.size - 1]
+ oldContent.forEach {
+ if (it == lastLine) {
+ sb.append(it)
+ } else {
+ sb.appendLine(it)
+ }
+ }
+ file.writeText(sb.toString())
+ }
+ file.appendText("""
+static_ip: ${ipaddr}
+redis_ip: ${redis}:6379
+project_id: ${proj}
+registry: ${registry}
+static_ip_name: ${ipaddrname}
+tag: $d_tag
+ """)
+ }
+}
+helm {
+ val backend by charts.creating {
+ chartName.set("backend")
+ sourceDir.set(file("../infrastructure/helm-backend"))
+ }
+ releases {
+ create("backend") {
+ from(backend)
+ }
+ }
+}
+task ("gkebackend") {
+ group = "deploy"
+ val init = tasks.getByName("terraformInit")
+ val apply = tasks.getByName("terraformApplyInf")
+ val takeConfig = tasks.getByName("takeConfig")
+ val push = tasks.getByName("pushBack")
+ val helm = tasks.getByName("helmInstallBackend")
+ val prepare = tasks.getByName("prepareConfig")
+ dependsOn(init)
+ dependsOn(apply)
+ dependsOn(takeConfig)
+ dependsOn(push)
+ dependsOn(helm)
+ dependsOn(prepare)
+ apply.mustRunAfter(init)
+ takeConfig.mustRunAfter(apply)
+ push.mustRunAfter(takeConfig)
+ helm.mustRunAfter(push)
Review Comment:
May we consider https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/helm? The helm provider is one of the community supported and doesn't come with the same support as the https://cloud.google.com/build/docs/cloud-builders#supported_builder_images_provided_by? Alternatively, there is https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/gke-deploy instead that provides the same variable substitution goals as helm. An example is https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/gke-deploy/doc/app/cloudbuild-with-configs.yaml#L29-L38
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AKosolapov commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
AKosolapov commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r990196554
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisite:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install GCloud to your Operation system
Review Comment:
I would rephrase: How to install gcloud CLI to your local environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] eantyshev commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
eantyshev commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r975219663
##########
.github/workflows/build_playground_backend_kubernetes.yml:
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: build_playground_backend_kubernetes
+on:
+ workflow_dispatch:
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ K8S_NAMESPACE: playground-backend
+ HELM_APP_NAME: playground-backend
+ DOCKERTAG: beta
+jobs:
+ deployBackend:
+ name: Start to Deploy
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ - name: Install kubectl
+ run: |
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+ - name: Install helm
+ run: |
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+ - name: Set up Cloud SDK
+ uses: google-github-actions/setup-gcloud@v0
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
+ which gcloud
+ gcloud auth activate-service-account --project=${{ secrets.GCP_PLAYGROUND_PROJECT_ID }} --key-file=/tmp/gcp_access.json
+ - name: create static ip
+ run: gcloud compute addresses create ${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}-static --global
+ - name: Update variables
+ run: |
+ echo "redis_ip: $(gcloud redis instances list --region us-central1 --format="value(host)"):6379" >> playground/infrastructure/helm-backend/values.yaml
+ echo "project_id: "${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "registry: "${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip: $(gcloud compute addresses list --filter "NAME:${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}-static" --format="value(address)")" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip_name: ${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}-static" >> playground/infrastructure/helm-backend/values.yaml
+ cat playground/infrastructure/helm-backend/values.yaml
+ - name: Get K8s Config
+ run: gcloud container clusters get-credentials --region us-central1-a playground-examples
+ - name: Login to Docker Registry
+ run: |
+ cat /tmp/gcp_access.json | docker login -u _json_key --password-stdin https://us-central1-docker.pkg.dev
+ - name: Build And Push Backend
+ run: ./gradlew playground:backend:containers:java:dockerTagPush -Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pbase-image='apache/beam_java8_sdk:${{ env.BEAM_VERSION }}' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Go Backend
+ run: ./gradlew playground:backend:containers:go:dockerTagPush -Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Python Backend
+ run: ./gradlew playground:backend:containers:python:dockerTagPush -Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Scio Backend
+ run: ./gradlew playground:backend:containers:scio:dockerTagPush -Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Router Backend
+ run: ./gradlew playground:backend:containers:router:dockerTagPush -Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Database and index creation
+ run: |
+ gcloud app create --region=us-central
+ gcloud datastore databases create --region=us-central
+ gcloud app deploy playground/index.yaml --project=${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}
+ - name: Backend setup
+ run: helm install backend playground/infrastructure/helm-backend
Review Comment:
Because we don't want to hide unsuccessful k8s deployments behind "success" workflow result
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r993697389
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) instructions for Linux operation systems
+* [Windows link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/) instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux operation systems
Review Comment:
Fixed
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) instructions for Linux operation systems
+* [Windows link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/) instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux operation systems
-## 3. Deploy application
+5. How to install Docker
+* [Docker link](https://docs.docker.com/engine/install/) instructions for Windows and Linux operation systems
-To deploy application run following steps:
+6. How to install GCloud
+* [Gcloud link](https://cloud.google.com/sdk/docs/install#linux) instructions for Windows and Linux operation systems
-* Authinticate in Artifact registry
+7. How to install Terraform
+* [Terraform link](https://www.terraform.io/downloads)
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r993694703
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989113028
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
+* Service account with the following roles should be created:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
+* Following APIs should be enabled:
+ - Identity and Access Management (IAM)
+ - Compute Engine API
+ - App Engine Admin API
+ - Cloud Resource Manager API
+
+* Make necessary changes in `playground/terraform/environment/beta/terraform.tfvars` file:
```
-
-## 1. Create new environment
-
-To provide information about the terraform backend, run the following commands
-
-* New environment folder
-
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+network_name = "network_name" #Choose network name
+project_id = "project_id" #Input project ID
+gke_name = "playground-backend" #Define GKE name
+region = "us-east1" #Choose region
+pg_location = "us-east1-b" #Choose location (should be in the region)
+state_bucket = "bucket_name" #Input bucket name
+bucket_examples_name = "bucket_name-example" #Input example bucket name
```
-
-* Backend config
-
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
+* Make necessary changes in `playground/terraform/environment/beta/state.tfbackend` file:
```
-
-* Terraform variables config and provide necessary variables
-
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
+bucket = "bucket_name" #input bucket name (will be used for tfstate file)
```
-
-Then provide necessary variables.
-
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
+* Export GOOGLE_APPLICATION_CREDENTIALS using following command:
```
-
-## 3. Deploy application
-
-To deploy application run following steps:
-
-* Authinticate in Artifact registry
-
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+ export GOOGLE_APPLICATION_CREDENTIALS=`your json key locaton`
```
-
-* Вeploy backend services
-
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+* Activate created service account using following command:
+```
+ gcloud auth activate-service-account `full principal service account` --key-file=`your json key locaton`
+```
+* Install kubectl:
+```
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+```
+* Install Helm:
+```
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+```
+# Infrastructure deployment:
+* Run following command for infrastructure deployment (please be sure that you are in the "beam" folder):
+```
+./gradlew playground:terraform:InitInfrastructure -Pproject_environment="beta"
+```
+# Backend deployment:
+* Login to Docker registry:
+```
+ cat `your json key locaton` | docker login -u _json_key --password-stdin https://`chosen_region`-docker.pkg.dev
+```
+* Login to GKE
+```
+gcloud container clusters get-credentials --region `chosen_pg_location` `gke_name` --project `project_id`
+```
+* Database index creation:
+```
+gcloud app deploy playground/index.yaml --project=`project_id`
+```
+* Run following command for backend deployment (please be sure that you are in the "beam" folder):
+```
+./gradlew playground:terraform:gkebackend -Pproject_environment="beta" -Pdocker-tag="beta"
Review Comment:
Yes. we can use different tag for images (when we will start to create images for Example check)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r993688704
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r992096779
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
Review Comment:
Fixed.
Add a new module which enable required APIs for BEAM Playground
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] eantyshev commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
eantyshev commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r975217635
##########
.github/workflows/build_playground_backend_kubernetes.yml:
##########
@@ -0,0 +1,85 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: build_playground_backend_kubernetes
+on:
+ workflow_dispatch:
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ K8S_NAMESPACE: playground-backend
+ HELM_APP_NAME: playground-backend
+ DOCKERTAG: beta
+jobs:
+ deployBackend:
+ name: Start to Deploy
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ - name: Install kubectl
+ run: |
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+ - name: Install helm
+ run: |
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+ - name: Set up Cloud SDK
+ uses: google-github-actions/setup-gcloud@v0
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
+ which gcloud
+ gcloud auth activate-service-account --project=${{ secrets.GCP_PLAYGROUND_PROJECT_ID }} --key-file=/tmp/gcp_access.json
+ - name: create static ip
+ run: gcloud compute addresses create ${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}-static --global
+ - name: Update variables
+ run: |
+ echo "redis_ip: $(gcloud redis instances list --region us-central1 --format="value(host)"):6379" >> playground/infrastructure/helm-backend/values.yaml
+ echo "project_id: "${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "registry: "${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip: $(gcloud compute addresses list --filter "NAME:${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}-static" --format="value(address)")" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip_name: ${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}-static" >> playground/infrastructure/helm-backend/values.yaml
+ cat playground/infrastructure/helm-backend/values.yaml
+ - name: Get K8s Config
+ run: gcloud container clusters get-credentials --region us-central1-a playground-examples
+ - name: Login to Docker Registry
+ run: |
+ cat /tmp/gcp_access.json | docker login -u _json_key --password-stdin https://us-central1-docker.pkg.dev
+ - name: Build And Push Backend
+ run: ./gradlew playground:backend:containers:java:dockerTagPush -Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pbase-image='apache/beam_java8_sdk:${{ env.BEAM_VERSION }}' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Go Backend
+ run: ./gradlew playground:backend:containers:go:dockerTagPush -Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Python Backend
+ run: ./gradlew playground:backend:containers:python:dockerTagPush -Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Scio Backend
+ run: ./gradlew playground:backend:containers:scio:dockerTagPush -Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Router Backend
+ run: ./gradlew playground:backend:containers:router:dockerTagPush -Pdocker-repository-root='${{ secrets.PLAYGROUND_REGISTRY_NAME}}/${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Database and index creation
+ run: |
+ gcloud app create --region=us-central
+ gcloud datastore databases create --region=us-central
+ gcloud app deploy playground/index.yaml --project=${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}
+ - name: Backend setup
+ run: helm install backend playground/infrastructure/helm-backend
Review Comment:
Do we need to verify if everything was brought up successfully via kubectl?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
olehborysevych commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r969737550
##########
playground/infrastructure/helm-backend/templates/go.yml:
##########
@@ -0,0 +1,30 @@
+apiVersion: apps/v1
Review Comment:
please rename deployment files to something like "deployment-go"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989095913
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
+* Service account with the following roles should be created:
Review Comment:
No we don't
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989099467
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
+* Service account with the following roles should be created:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
+* Following APIs should be enabled:
+ - Identity and Access Management (IAM)
+ - Compute Engine API
+ - App Engine Admin API
+ - Cloud Resource Manager API
+
+* Make necessary changes in `playground/terraform/environment/beta/terraform.tfvars` file:
```
-
-## 1. Create new environment
-
-To provide information about the terraform backend, run the following commands
-
-* New environment folder
-
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+network_name = "network_name" #Choose network name
+project_id = "project_id" #Input project ID
+gke_name = "playground-backend" #Define GKE name
+region = "us-east1" #Choose region
+pg_location = "us-east1-b" #Choose location (should be in the region)
+state_bucket = "bucket_name" #Input bucket name
+bucket_examples_name = "bucket_name-example" #Input example bucket name
```
-
-* Backend config
-
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
+* Make necessary changes in `playground/terraform/environment/beta/state.tfbackend` file:
```
-
-* Terraform variables config and provide necessary variables
-
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
+bucket = "bucket_name" #input bucket name (will be used for tfstate file)
```
-
-Then provide necessary variables.
-
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
+* Export GOOGLE_APPLICATION_CREDENTIALS using following command:
```
-
-## 3. Deploy application
-
-To deploy application run following steps:
-
-* Authinticate in Artifact registry
-
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+ export GOOGLE_APPLICATION_CREDENTIALS=`your json key locaton`
```
-
-* Вeploy backend services
-
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+* Activate created service account using following command:
+```
+ gcloud auth activate-service-account `full principal service account` --key-file=`your json key locaton`
Review Comment:
https://cloud.google.com/container-registry/docs/advanced-authentication
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989102363
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
+* Service account with the following roles should be created:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
+* Following APIs should be enabled:
+ - Identity and Access Management (IAM)
+ - Compute Engine API
+ - App Engine Admin API
+ - Cloud Resource Manager API
+
+* Make necessary changes in `playground/terraform/environment/beta/terraform.tfvars` file:
```
-
-## 1. Create new environment
-
-To provide information about the terraform backend, run the following commands
-
-* New environment folder
-
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+network_name = "network_name" #Choose network name
+project_id = "project_id" #Input project ID
+gke_name = "playground-backend" #Define GKE name
+region = "us-east1" #Choose region
+pg_location = "us-east1-b" #Choose location (should be in the region)
+state_bucket = "bucket_name" #Input bucket name
+bucket_examples_name = "bucket_name-example" #Input example bucket name
```
-
-* Backend config
-
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
+* Make necessary changes in `playground/terraform/environment/beta/state.tfbackend` file:
```
-
-* Terraform variables config and provide necessary variables
-
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
+bucket = "bucket_name" #input bucket name (will be used for tfstate file)
```
-
-Then provide necessary variables.
-
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
+* Export GOOGLE_APPLICATION_CREDENTIALS using following command:
```
-
-## 3. Deploy application
-
-To deploy application run following steps:
-
-* Authinticate in Artifact registry
-
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+ export GOOGLE_APPLICATION_CREDENTIALS=`your json key locaton`
Review Comment:
path could be "any" - absolute or relative
Link to "How create an account"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989102956
##########
playground/terraform/environment/beta/state.tfbackend:
##########
@@ -17,4 +17,4 @@
# under the License.
#
-bucket = "beam_playground_terraform"
+bucket = "pg-second"
Review Comment:
any name, doesn't matter. It was in "readme"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r991868979
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] eantyshev commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
eantyshev commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r991079524
##########
playground/terraform/build.gradle.kts:
##########
@@ -395,3 +372,111 @@ task("deployBackend") {
dependsOn(deploy)
}
+task("takeConfig") {
+ group = "deploy"
+ doLast {
+ var ipaddr = ""
+ var redis = ""
+ var proj = ""
+ var registry = ""
+ var ipaddrname = ""
+ var d_tag = ""
+ var stdout = ByteArrayOutputStream()
+ if (project.hasProperty("docker-tag")) {
+ d_tag = project.property("docker-tag") as String
+ }
+ exec {
+ commandLine = listOf("terraform", "output", "playground_static_ip_address")
+ standardOutput = stdout
+ }
+ ipaddr = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
Review Comment:
NIT this line belongs to `exec` statement below, not above, so let's group it more naturally
##########
playground/terraform/build.gradle.kts:
##########
@@ -395,3 +372,111 @@ task("deployBackend") {
dependsOn(deploy)
}
+task("takeConfig") {
Review Comment:
NIT maybe, `setHelmValues`?
##########
playground/terraform/build.gradle.kts:
##########
@@ -395,3 +372,111 @@ task("deployBackend") {
dependsOn(deploy)
}
+task("takeConfig") {
+ group = "deploy"
+ doLast {
+ var ipaddr = ""
+ var redis = ""
+ var proj = ""
+ var registry = ""
+ var ipaddrname = ""
+ var d_tag = ""
+ var stdout = ByteArrayOutputStream()
+ if (project.hasProperty("docker-tag")) {
+ d_tag = project.property("docker-tag") as String
+ }
+ exec {
+ commandLine = listOf("terraform", "output", "playground_static_ip_address")
+ standardOutput = stdout
+ }
+ ipaddr = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+
+ exec {
+ commandLine = listOf("terraform", "output", "playground_redis_ip")
+ standardOutput = stdout
+ }
+ redis = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "playground_gke_project")
+ standardOutput = stdout
+ }
+ proj = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "docker-repository-root")
+ standardOutput = stdout
+ }
+ registry = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "playground_static_ip_address_name")
+ standardOutput = stdout
+ }
+ ipaddrname = stdout.toString().trim().replace("\"", "")
Review Comment:
NIT Do we have functions to wrap this repetitive statements?
##########
playground/terraform/build.gradle.kts:
##########
@@ -395,3 +372,111 @@ task("deployBackend") {
dependsOn(deploy)
}
+task("takeConfig") {
+ group = "deploy"
+ doLast {
+ var ipaddr = ""
+ var redis = ""
+ var proj = ""
+ var registry = ""
+ var ipaddrname = ""
+ var d_tag = ""
+ var stdout = ByteArrayOutputStream()
+ if (project.hasProperty("docker-tag")) {
+ d_tag = project.property("docker-tag") as String
+ }
+ exec {
+ commandLine = listOf("terraform", "output", "playground_static_ip_address")
+ standardOutput = stdout
+ }
+ ipaddr = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+
+ exec {
+ commandLine = listOf("terraform", "output", "playground_redis_ip")
+ standardOutput = stdout
+ }
+ redis = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "playground_gke_project")
+ standardOutput = stdout
+ }
+ proj = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "docker-repository-root")
+ standardOutput = stdout
+ }
+ registry = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "playground_static_ip_address_name")
+ standardOutput = stdout
+ }
+ ipaddrname = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
Review Comment:
NIT seems like this isn't needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damondouglas commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
damondouglas commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r993912780
##########
playground/infrastructure/helm-backend/templates/ingress.yml:
##########
@@ -0,0 +1,79 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: ingress-playground
+ annotations:
+ kubernetes.io/ingress.global-static-ip-name: {{ .Values.static_ip_name }}
+ networking.gke.io/managed-certificates: {{ .Values.project_id }}
+ networking.gke.io/v1beta1.FrontendConfig: pg-frontend
+ ingress.kubernetes.io/enable-cors: "true"
+ labels:
+ app: playground-go
+spec:
+ rules:
+ - host: "go.{{ .Values.static_ip }}.nip.io"
+ http:
+ paths:
+ - path: "/*"
+ pathType: ImplementationSpecific
+ backend:
+ service:
+ name: playground-go
+ port:
+ number: {{ .Values.service.port }}
+ - host: "java.{{ .Values.static_ip }}.nip.io"
Review Comment:
May need security review/approval.
##########
.github/workflows/playground_deploy_infrastructure.yml:
##########
@@ -0,0 +1,18 @@
+name: New environment creation
+on:
+ workflow_dispatch:
+jobs:
+ Infrastructure_deployment:
+ runs-on: ubuntu-latest
+ env:
+ PROJECT_ENV: beta
+ steps:
+ - name: git clone repo
+ uses: actions/checkout@v2
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.S3D_GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
Review Comment:
uses your secret S3D
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,114 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. [How to create a new project in Google Cloud Platform](https://cloud.google.com/resource-manager/docs/creating-managing-projects/) _(Note: How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects)_
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. [How to create a new service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) _(Note: You will find the instruction "How to create Service account" for your new project)_
-# Deployment steps
+3. [How to create a JSON key for a Google Service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) _(Note: In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication)_
-## 0. Create GCS bucket for state
+4. [How to create a new bucket](https://cloud.google.com/storage/docs/creating-buckets) _(Note: Short instruction for bucket creation. You will need at least one bucket for file keeping)_
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
-```
+5. [How to assign a new role for the service account](https://cloud.google.com/iam/docs/granting-changing-revoking-access) _(Note: It will explain how to assign required roles for your service account)_
-## 1. Create new environment
+6. [How to install gcloud CLI to your local environment](https://cloud.google.com/sdk/docs/install?hl=en) _(Note: GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)_
-To provide information about the terraform backend, run the following commands
+7. [What are Google Cloud Platform APIs](https://cloud.google.com/apis) _(Note: Short description of all Google Platform APIs)_
-* New environment folder
+8. [Google Cloud Platform naming policy](https://cloud.google.com/compute/docs/naming-resources) _(Note: Describes the naming convention for Compute Engine resources)_
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
-```
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-* Backend config
+***Operation System preparation steps:***
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-* Terraform variables config and provide necessary variables
+* [Java](https://adoptopenjdk.net/)
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+* [Kubernetes Command Line Interface](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
-Then provide necessary variables.
+* [HELM](https://helm.sh/docs/intro/install/)
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+* [Docker](https://docs.docker.com/engine/install/)
-## 3. Deploy application
+* [Terraform](https://www.terraform.io/downloads)
-To deploy application run following steps:
+After installations of the required packages will be completed, you will need to download ***BEAM Playground*** from the repository.
-* Authinticate in Artifact registry
+Once downloading is completed you will find the "beam" folder on your disk (you can execute the `ls -la` command for Linux, or the `dir` command for Windows). Open that folder
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+It is possible to configure BEAM to work with the different versions of the environment (like staging, production, etc.), you will need to create a special folder in `playground/terraform/environment` and put all configuration files inside:
+* File name - `terraform.tfvars`, items inside:
```
+network_name = "network_name" #Enter the network name - you can choose any name for the network according to the naming policy above
+project_id = "project_id" #Enter the project ID - ID of created project
+gke_name = "playground-backend" #Set the GKE name - you can choose any name for Google Kubernetes Engine according to the naming policy above
+region = "us-east1" #Set the region - preferred region according to your needs
+location = "us-east1-b" #Select the location - location should be in region you set before
+state_bucket = "bucket_name" #Name of bucket - Google Cloud bucket where BEAM Playground will put temp files, [terraform state file] (https://spacelift.io/blog/terraform-state)
+bucket_examples_name = "bucket_name-example" #Enter an example bucket name - bucket for some build-in examples for BEAM Playground
+```
+* File name - `state.tfbackend`, item inside:
+```
+bucket = "bucket_name" #input bucket name - will be used for terraform tfstate file
+```
+Then, let's configure authentication for the Google Cloud Platform:
-* Вeploy backend services
-
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+* The following command allows us to authenticate using JSON key file
+```
+ export GOOGLE_APPLICATION_CREDENTIALS=`your service account JSON key location` (absolute path)
+```
+* Using the following command, we will activate the newly created service account:
+```
+ gcloud auth activate-service-account `full principal service account` --key-file=`your service account JSON key location` (absolute path)
```
-* Deploy frontend service
+# Infrastructure deployment:
+* To deploy the Infrastructure, use the following command (please be sure that you are in the "beam" folder):
+```
+./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env" (env - folder name which you created for configuration files)
+```
+# Backend deployment:
+Once the script was executed successfully, you will need to authenticate on Docker and Google Kubernetes Engine
+The following command will authorize us in the Docker registry
+```
+ cat `your service account json key locaton` | docker login -u _json_key --password-stdin https://`chosen_region`-docker.pkg.dev
+```
+* The following command will authenticate us in GKE
+```
+gcloud container clusters get-credentials --region `chosen_pg_location` `gke_name` --project `project_id`
+```
+* We need to create database indexes for BEAM playground examples by the following command:
+```
+gcloud app deploy playground/index.yaml --project=`project_id`
+```
+That's all, the configuration of the environment has been completed. For deploying the backend part to the Google cloud Kubernetes engine, please execute the following command (Ensure you are in the "beam" folder):
+```
+./gradlew playground:terraform:gkebackend -Pproject_environment="env" -Pdocker-tag="tag" (env - folder name which you created for configuration files, tag - image tag for backend)
Review Comment:
Getting error:
```
Caused by: java.io.IOException: Cannot run program "helm" (in directory "<path to project>/playground/terraform"): error=2, No such file or directory
at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)
... 6 more
Caused by: java.io.IOException: error=2, No such file or directory
... 7 more
```
When running, `helm version`, I see:
```
version.BuildInfo{Version:"v3.9.4", GitCommit:"dbc6d8e20fe1d58d50e6ed30f09a04a77e4c68db", GitTreeState:"clean", GoVersion:"go1.17.13"}
```
##########
playground/infrastructure/helm-backend/templates/ingress.yml:
##########
@@ -0,0 +1,79 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: ingress-playground
+ annotations:
+ kubernetes.io/ingress.global-static-ip-name: {{ .Values.static_ip_name }}
+ networking.gke.io/managed-certificates: {{ .Values.project_id }}
+ networking.gke.io/v1beta1.FrontendConfig: pg-frontend
+ ingress.kubernetes.io/enable-cors: "true"
+ labels:
+ app: playground-go
+spec:
+ rules:
+ - host: "go.{{ .Values.static_ip }}.nip.io"
+ http:
+ paths:
+ - path: "/*"
+ pathType: ImplementationSpecific
+ backend:
+ service:
+ name: playground-go
+ port:
+ number: {{ .Values.service.port }}
+ - host: "java.{{ .Values.static_ip }}.nip.io"
+ http:
+ paths:
+ - path: "/*"
+ pathType: ImplementationSpecific
+ backend:
+ service:
+ name: playground-java
+ port:
+ number: {{ .Values.service.port }}
+ - host: "python.{{ .Values.static_ip }}.nip.io"
Review Comment:
May need security review/approval.
##########
playground/infrastructure/helm-backend/templates/ingress.yml:
##########
@@ -0,0 +1,79 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: ingress-playground
+ annotations:
+ kubernetes.io/ingress.global-static-ip-name: {{ .Values.static_ip_name }}
+ networking.gke.io/managed-certificates: {{ .Values.project_id }}
+ networking.gke.io/v1beta1.FrontendConfig: pg-frontend
+ ingress.kubernetes.io/enable-cors: "true"
+ labels:
+ app: playground-go
+spec:
+ rules:
+ - host: "go.{{ .Values.static_ip }}.nip.io"
+ http:
+ paths:
+ - path: "/*"
+ pathType: ImplementationSpecific
+ backend:
+ service:
+ name: playground-go
+ port:
+ number: {{ .Values.service.port }}
+ - host: "java.{{ .Values.static_ip }}.nip.io"
+ http:
+ paths:
+ - path: "/*"
+ pathType: ImplementationSpecific
+ backend:
+ service:
+ name: playground-java
+ port:
+ number: {{ .Values.service.port }}
+ - host: "python.{{ .Values.static_ip }}.nip.io"
+ http:
+ paths:
+ - path: "/*"
+ pathType: ImplementationSpecific
+ backend:
+ service:
+ name: playground-python
+ port:
+ number: {{ .Values.service.port }}
+ - host: "router.{{ .Values.static_ip }}.nip.io"
Review Comment:
May need security review/approval.
##########
playground/infrastructure/helm-backend/templates/managecert.yml:
##########
@@ -0,0 +1,27 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: networking.gke.io/v1
+kind: ManagedCertificate
+metadata:
+ name: {{ .Values.project_id }}
+spec:
+ domains:
+ - go.{{ .Values.static_ip }}.nip.io
+ - java.{{ .Values.static_ip }}.nip.io
+ - python.{{ .Values.static_ip }}.nip.io
+ - router.{{ .Values.static_ip }}.nip.io
+ - scio.{{ .Values.static_ip }}.nip.io
Review Comment:
May need security review/approval.
##########
playground/infrastructure/helm-backend/templates/ingress.yml:
##########
@@ -0,0 +1,79 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: ingress-playground
+ annotations:
+ kubernetes.io/ingress.global-static-ip-name: {{ .Values.static_ip_name }}
+ networking.gke.io/managed-certificates: {{ .Values.project_id }}
+ networking.gke.io/v1beta1.FrontendConfig: pg-frontend
+ ingress.kubernetes.io/enable-cors: "true"
+ labels:
+ app: playground-go
+spec:
+ rules:
+ - host: "go.{{ .Values.static_ip }}.nip.io"
+ http:
+ paths:
+ - path: "/*"
+ pathType: ImplementationSpecific
+ backend:
+ service:
+ name: playground-go
+ port:
+ number: {{ .Values.service.port }}
+ - host: "java.{{ .Values.static_ip }}.nip.io"
+ http:
+ paths:
+ - path: "/*"
+ pathType: ImplementationSpecific
+ backend:
+ service:
+ name: playground-java
+ port:
+ number: {{ .Values.service.port }}
+ - host: "python.{{ .Values.static_ip }}.nip.io"
+ http:
+ paths:
+ - path: "/*"
+ pathType: ImplementationSpecific
+ backend:
+ service:
+ name: playground-python
+ port:
+ number: {{ .Values.service.port }}
+ - host: "router.{{ .Values.static_ip }}.nip.io"
+ http:
+ paths:
+ - path: "/*"
+ pathType: ImplementationSpecific
+ backend:
+ service:
+ name: playground-router
+ port:
+ number: {{ .Values.service.port }}
+ - host: "scio.{{ .Values.static_ip }}.nip.io"
Review Comment:
May need security review/approval.
##########
playground/infrastructure/helm-backend/templates/ingress.yml:
##########
@@ -0,0 +1,79 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: ingress-playground
+ annotations:
+ kubernetes.io/ingress.global-static-ip-name: {{ .Values.static_ip_name }}
+ networking.gke.io/managed-certificates: {{ .Values.project_id }}
+ networking.gke.io/v1beta1.FrontendConfig: pg-frontend
+ ingress.kubernetes.io/enable-cors: "true"
+ labels:
+ app: playground-go
+spec:
+ rules:
+ - host: "go.{{ .Values.static_ip }}.nip.io"
Review Comment:
@pabloem May need security review/approval. I've flagged the others below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r994219442
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,114 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. [How to create a new project in Google Cloud Platform](https://cloud.google.com/resource-manager/docs/creating-managing-projects/) _(Note: How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects)_
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. [How to create a new service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) _(Note: You will find the instruction "How to create Service account" for your new project)_
-# Deployment steps
+3. [How to create a JSON key for a Google Service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) _(Note: In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication)_
-## 0. Create GCS bucket for state
+4. [How to create a new bucket](https://cloud.google.com/storage/docs/creating-buckets) _(Note: Short instruction for bucket creation. You will need at least one bucket for file keeping)_
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
-```
+5. [How to assign a new role for the service account](https://cloud.google.com/iam/docs/granting-changing-revoking-access) _(Note: It will explain how to assign required roles for your service account)_
-## 1. Create new environment
+6. [How to install gcloud CLI to your local environment](https://cloud.google.com/sdk/docs/install?hl=en) _(Note: GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)_
-To provide information about the terraform backend, run the following commands
+7. [What are Google Cloud Platform APIs](https://cloud.google.com/apis) _(Note: Short description of all Google Platform APIs)_
-* New environment folder
+8. [Google Cloud Platform naming policy](https://cloud.google.com/compute/docs/naming-resources) _(Note: Describes the naming convention for Compute Engine resources)_
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
-```
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-* Backend config
+***Operation System preparation steps:***
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-* Terraform variables config and provide necessary variables
+* [Java](https://adoptopenjdk.net/)
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+* [Kubernetes Command Line Interface](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
-Then provide necessary variables.
+* [HELM](https://helm.sh/docs/intro/install/)
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+* [Docker](https://docs.docker.com/engine/install/)
-## 3. Deploy application
+* [Terraform](https://www.terraform.io/downloads)
-To deploy application run following steps:
+After installations of the required packages will be completed, you will need to download ***BEAM Playground*** from the repository.
-* Authinticate in Artifact registry
+Once downloading is completed you will find the "beam" folder on your disk (you can execute the `ls -la` command for Linux, or the `dir` command for Windows). Open that folder
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+It is possible to configure BEAM to work with the different versions of the environment (like staging, production, etc.), you will need to create a special folder in `playground/terraform/environment` and put all configuration files inside:
+* File name - `terraform.tfvars`, items inside:
```
+network_name = "network_name" #Enter the network name - you can choose any name for the network according to the naming policy above
+project_id = "project_id" #Enter the project ID - ID of created project
+gke_name = "playground-backend" #Set the GKE name - you can choose any name for Google Kubernetes Engine according to the naming policy above
+region = "us-east1" #Set the region - preferred region according to your needs
+location = "us-east1-b" #Select the location - location should be in region you set before
+state_bucket = "bucket_name" #Name of bucket - Google Cloud bucket where BEAM Playground will put temp files, [terraform state file] (https://spacelift.io/blog/terraform-state)
+bucket_examples_name = "bucket_name-example" #Enter an example bucket name - bucket for some build-in examples for BEAM Playground
+```
+* File name - `state.tfbackend`, item inside:
+```
+bucket = "bucket_name" #input bucket name - will be used for terraform tfstate file
+```
+Then, let's configure authentication for the Google Cloud Platform:
-* Вeploy backend services
-
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+* The following command allows us to authenticate using JSON key file
+```
+ export GOOGLE_APPLICATION_CREDENTIALS=`your service account JSON key location` (absolute path)
+```
+* Using the following command, we will activate the newly created service account:
+```
+ gcloud auth activate-service-account `full principal service account` --key-file=`your service account JSON key location` (absolute path)
```
-* Deploy frontend service
+# Infrastructure deployment:
+* To deploy the Infrastructure, use the following command (please be sure that you are in the "beam" folder):
+```
+./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env" (env - folder name which you created for configuration files)
+```
+# Backend deployment:
+Once the script was executed successfully, you will need to authenticate on Docker and Google Kubernetes Engine
+The following command will authorize us in the Docker registry
+```
+ cat `your service account json key locaton` | docker login -u _json_key --password-stdin https://`chosen_region`-docker.pkg.dev
+```
+* The following command will authenticate us in GKE
+```
+gcloud container clusters get-credentials --region `chosen_pg_location` `gke_name` --project `project_id`
+```
+* We need to create database indexes for BEAM playground examples by the following command:
+```
+gcloud app deploy playground/index.yaml --project=`project_id`
+```
+That's all, the configuration of the environment has been completed. For deploying the backend part to the Google cloud Kubernetes engine, please execute the following command (Ensure you are in the "beam" folder):
+```
+./gradlew playground:terraform:gkebackend -Pproject_environment="env" -Pdocker-tag="tag" (env - folder name which you created for configuration files, tag - image tag for backend)
Review Comment:
Hello, Damon
I hope you're switching branches from the master to the current one. Because, yes, till we merge it to the branch, the helm folder will not be there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
olehborysevych commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r992007742
##########
.github/workflows/playground_deploy_backend_gke.yml:
##########
@@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+name: kuber-bkend
+on:
+ workflow_dispatch:
+
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ DOCKERTAG: beta
+jobs:
+ deployBackend:
+ name: init
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ - name: Install kubectl
+ run: |
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+ - name: Install helm
+ run: |
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+ - name: Set up Cloud SDK
+ uses: google-github-actions/setup-gcloud@v0
+ - name: Setup GCP account
+ run: |
+ echo "${{ secrets.S3D_GCP_PLAYGROUND_SA_KEY }}" | base64 -d > /tmp/gcp_access.json
+ which gcloud
+ gcloud auth activate-service-account --project=${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }} --key-file=/tmp/gcp_access.json
+ - name: Update variables
+ run: |
+ echo "redis_ip: $(gcloud redis instances list --region us-central1 --format="value(host)"):6379" >> playground/infrastructure/helm-backend/values.yaml
+ echo "project_id: "${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "registry: "${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository"" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip: $(gcloud compute addresses list --filter "NAME:${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static" --format="value(address)")" >> playground/infrastructure/helm-backend/values.yaml
+ echo "static_ip_name: ${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}-static" >> playground/infrastructure/helm-backend/values.yaml
+ cat playground/infrastructure/helm-backend/values.yaml
+ - name: Get K8s Config
+ run: gcloud container clusters get-credentials --region us-central1-a playground-examples
+ - name: Login to Docker Registry
+ run: |
+ cat /tmp/gcp_access.json | docker login -u _json_key --password-stdin https://us-central1-docker.pkg.dev
+ - name: Build And Push Backend
+ run: ./gradlew playground:backend:containers:java:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pbase-image='apache/beam_java8_sdk:${{ env.BEAM_VERSION }}' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Go Backend
+ run: ./gradlew playground:backend:containers:go:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Python Backend
+ run: ./gradlew playground:backend:containers:python:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Scio Backend
+ run: ./gradlew playground:backend:containers:scio:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Build And Push Router Backend
+ run: ./gradlew playground:backend:containers:router:dockerTagPush -Pdocker-repository-root='${{ secrets.S3D_PLAYGROUND_REGISTRY_NAME}}/${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}/playground-repository' -Pdocker-tag=${{ env.DOCKERTAG }}
+ - name: Index creation
+ run: |
+ gcloud app deploy playground/index.yaml --project=${{ secrets.S3D_GCP_PLAYGROUND_PROJECT_ID }}
Review Comment:
@MakarkinSAkvelon S3D_* keys are still here. Are we going to leave them for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r993690494
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
Review Comment:
Yes, we need a bucket for tfstate file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989102363
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
+* Service account with the following roles should be created:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
+* Following APIs should be enabled:
+ - Identity and Access Management (IAM)
+ - Compute Engine API
+ - App Engine Admin API
+ - Cloud Resource Manager API
+
+* Make necessary changes in `playground/terraform/environment/beta/terraform.tfvars` file:
```
-
-## 1. Create new environment
-
-To provide information about the terraform backend, run the following commands
-
-* New environment folder
-
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+network_name = "network_name" #Choose network name
+project_id = "project_id" #Input project ID
+gke_name = "playground-backend" #Define GKE name
+region = "us-east1" #Choose region
+pg_location = "us-east1-b" #Choose location (should be in the region)
+state_bucket = "bucket_name" #Input bucket name
+bucket_examples_name = "bucket_name-example" #Input example bucket name
```
-
-* Backend config
-
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
+* Make necessary changes in `playground/terraform/environment/beta/state.tfbackend` file:
```
-
-* Terraform variables config and provide necessary variables
-
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
+bucket = "bucket_name" #input bucket name (will be used for tfstate file)
```
-
-Then provide necessary variables.
-
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
+* Export GOOGLE_APPLICATION_CREDENTIALS using following command:
```
-
-## 3. Deploy application
-
-To deploy application run following steps:
-
-* Authinticate in Artifact registry
-
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+ export GOOGLE_APPLICATION_CREDENTIALS=`your json key locaton`
Review Comment:
path could be "any" - absolute or relative
Link to "How to create an account"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989104972
##########
playground/infrastructure/helm-backend/values.yaml:
##########
@@ -0,0 +1,36 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+replicaCount: 1
+image:
+ java_image: beam_playground-backend-java
+ go_image: beam_playground-backend-go
+ router_image: beam_playground-backend-router
+ scio_image: beam_playground-backend-scio
+ python_image: beam_playground-backend-python
+ pullPolicy: Always
+ tag: "beta"
Review Comment:
it's a variables in HELM, I don't think that we need to describe hos it's work, because they shouldn't be changed by hands
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AKosolapov commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
AKosolapov commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r990211885
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisite:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install GCloud to your Operation system
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+When you create a new project, you must add a service account with a JSON key and a bucket according to the instructions above. Also, according to the instructions, configure the roles for the service account:
Review Comment:
Consider rephrasing:
After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select `bucket_name` for storing the Terraform state data according to the instructions above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23208:
URL: https://github.com/apache/beam/pull/23208#issuecomment-1290460750
Reminder, please take a look at this pr: @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] eantyshev commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
eantyshev commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r975137467
##########
.github/workflows/k8e-backend.yml:
##########
@@ -0,0 +1,85 @@
+#
Review Comment:
It should replace the existing playground_deploy_backend.yaml
More to that, we are leaving many obsolete files behind, for example
playground/terraform/applications/backend/*
- We need to discover if there's anything valuable there and make the identical Helm conf
- Then we need to delete those terraform files if they're not relevant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23208:
URL: https://github.com/apache/beam/pull/23208#issuecomment-1252507536
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @damccorm for label build.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damondouglas commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
damondouglas commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r991451157
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
Review Comment:
Does this step refer to creating the terraform service account? Otherwise, aren't we using terraform to provision needed service accounts for the project?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
Review Comment:
Is this bucket needed for terraform state management? I see the "Create GCS bucket for state" deleted so it isn't clear now the purpose.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
Review Comment:
Where are service account keys needed?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
Review Comment:
Google Cloud APIs can be enabled via infrastructure-as-code. See https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_project_service. Typically, I use a list and the `for-each`. Due to the need for explicit dependency declaration in terraform with Google Cloud resources, I will provision the enabling of APIs in a separate pre-requesite terraform module prior to applying subsequent modules. Alternatively, one would need to remember to explicitly declare the dependency on the `google_project_service` enable block via `depends_on`.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
Review Comment:
Are these the IAM roles needed for a particular service account? If so,
1. It's recommended to bind IAM roles using infrastructure-as-code where feasible. What I've done in the past is create a setup module that provisions the terraform service account. I then bind a "developer email" which is the account authenticated via the gcloud commandline utility with the role https://cloud.google.com/iam/docs/service-accounts#token-creator-role. The https://cloud.google.com/iam/docs/service-accounts#user-role may be needed as well but I would try first with just the token creator role. Then I would provision the minimally permissive IAM roles to the terraform service account.
2. A note about the selected IAM roles above. There are times when admin roles are needed. However, for most applications, I would incrementally observe the minimally least permissive role that enables the service account to perform its needed job.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
Review Comment:
May we consider removing this and not delete the original docker pre-requisite? The link refers the developer to read more about Docker if it is not known to the developer.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
Review Comment:
I would provision these APIs using https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_project_service
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) instructions for Linux operation systems
+* [Windows link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/) instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows and Linux operation systems
Review Comment:
Perhaps, we list helm as one of the pre-requisites above?
```markdown
[helm](https://helm.sh/docs/intro/install/)
```
##########
playground/terraform/build.gradle.kts:
##########
@@ -395,3 +372,111 @@ task("deployBackend") {
dependsOn(deploy)
}
+task("takeConfig") {
+ group = "deploy"
+ doLast {
+ var ipaddr = ""
+ var redis = ""
+ var proj = ""
+ var registry = ""
+ var ipaddrname = ""
+ var d_tag = ""
+ var stdout = ByteArrayOutputStream()
+ if (project.hasProperty("docker-tag")) {
+ d_tag = project.property("docker-tag") as String
+ }
+ exec {
+ commandLine = listOf("terraform", "output", "playground_static_ip_address")
+ standardOutput = stdout
+ }
+ ipaddr = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+
+ exec {
+ commandLine = listOf("terraform", "output", "playground_redis_ip")
+ standardOutput = stdout
+ }
+ redis = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "playground_gke_project")
+ standardOutput = stdout
+ }
+ proj = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "docker-repository-root")
+ standardOutput = stdout
+ }
+ registry = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "playground_static_ip_address_name")
+ standardOutput = stdout
+ }
+ ipaddrname = stdout.toString().trim().replace("\"", "")
Review Comment:
Now that we see this type of need for extra code in a gradle context, should we consider not using gradle to wrap our terraform? I've had concerns initially about the gradle terraform wrapper. If I were to develop this automation, I would do two things to satisfy provisioning from one's local development environment manually as well as via automation. First, I would name the folders in order of their designed application. For example, I typically have a `00.setup` folder, followed by a `01.network` folder. These respective folders would hold the infrastructure-as-code needed as explicitly named by the folder. When automating infrastructure-as-code, in Google Cloud, we can make use of Cloud Build. https://github.com/GoogleCloudPlatform/solutions-terraform-cloudbuild-gitops provides an end-to-end example.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) instructions for Linux operation systems
+* [Windows link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/) instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux operation systems
-## 3. Deploy application
+5. How to install Docker
+* [Docker link](https://docs.docker.com/engine/install/) instructions for Windows and Linux operation systems
-To deploy application run following steps:
+6. How to install GCloud
+* [Gcloud link](https://cloud.google.com/sdk/docs/install#linux) instructions for Windows and Linux operation systems
-* Authinticate in Artifact registry
+7. How to install Terraform
+* [Terraform link](https://www.terraform.io/downloads)
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
-```
+After installations of the required packages will be completed, you will need to download ***BEAM Playground*** from the repository. Please open the terminal and execute the following command:
-* Вeploy backend services
+`git clone 'url_to_github_project_with_beam_playground'`
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+Once downloading is completed you will find the "beam" folder on your disk (you can execute the `ls -la` command for Linux, or the `dir` command for Windows). Open that folder
+
+It is possible to configure BEAM to work with the different versions of the environment (like staging, production, etc.), you will need to create a special folder in `playground/terraform/environment` and put all configuration files inside:
+* File name - `terraform.tfvars`, items inside:
+```
+network_name = "network_name" #Enter the network name - you can choose any name for the network according to the naming policy above
+project_id = "project_id" #Enter the project ID - ID of created project
+gke_name = "playground-backend" #Set the GKE name - you can choose any name for Google Kubernetes Engine according to the naming policy above
+region = "us-east1" #Set the region - preferred region according to your needs
+pg_location = "us-east1-b" #Select the location - location should be in region you set before
+state_bucket = "bucket_name" #Name of bucket - Google Cloud bucket where BEAM Playground will put temp files, [terraform state file] (https://spacelift.io/blog/terraform-state)
+bucket_examples_name = "bucket_name-example" #Enter an example bucket name - bucket for some build-in examples for BEAM Playground
+```
+* File name - `state.tfbackend`, item inside:
```
+bucket = "bucket_name" #input bucket name - will be used for terraform tfstate file
+```
+Then, let's configure authentication for the Google Cloud Platform:
-* Deploy frontend service
+* The following command allows us to authenticate using JSON key file
+```
+ export GOOGLE_APPLICATION_CREDENTIALS=`your service account JSON key location` (absolute path)
+```
+* Using the following command, we will activate the newly created service account:
+```
+ gcloud auth activate-service-account `full principal service account` --key-file=`your service account JSON key location` (absolute path)
+```
Review Comment:
Binding a developer's account (the account authenticated via `gcloud auth application-default login`) with the role Service Account Token Creator eliminates the need for the older and less secure practice of using JSON keys.
##########
playground/terraform/infrastructure/appengine/main.tf:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+resource "google_app_engine_application" "app_playground" {
+ project = var.project_id
+ location_id = var.region == "us-central1" ? var.location_id_us : var.region
Review Comment:
Can we just have:
```terraform
location_id = var.location_id
```
##########
playground/terraform/infrastructure/variables.tf:
##########
@@ -167,5 +165,8 @@ variable "gke_name" {
variable "gke_location" {
description = "Location of GKE cluster"
- default = "us-central1-a"
+}
+
+variable "pg_location" {
Review Comment:
Personally I found the `pg` in this `pg_location` name confusing initially. May we consider variable names in the language of the cloud environment? For example, `gke_location` is clear in that it refers to the location of the GKE node. In some infrastructure-as-code modules, I've even seen just the use of `region` as it is conventionally understood that region refers to the Compute Engine region. Then, when applying to the terraform GKE module, the location property is simply assigned to the `var.region` variable.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
Review Comment:
May we replace with:
```markdown
- [Google Cloud Project with billing enabled](https://cloud.google.com/resource-manager/docs/creating-managing-projects) _(Note: New temporary projects recommended for development and testing.)_
```
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
Review Comment:
Wouldn't a developer on their local machine require these?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
Review Comment:
This link already existed as part of the original listed pre-requisites.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
Review Comment:
Usually the terraform output provides naming-convention errors and/or the documentation provides the relevant naming convention.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
Review Comment:
Wouldn't we achieve this step in the infrastructure as code?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary instructions for Windows and Linux operation systems
Review Comment:
Should we just list this above in the pre-requisites:
```markdown
[Java 11](https://openjdk.org/projects/jdk/11/)
```
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) instructions for Linux operation systems
+* [Windows link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/) instructions for Windows operation systems
Review Comment:
What about Mac? Instead, shouldn't we just refer to the kubectl already bundled as an optional component of gcloud? https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl. Refering to this covers the supported operating systems without needed to install via kubernetes.io directly.
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) instructions for Linux operation systems
+* [Windows link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/) instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux operation systems
Review Comment:
May we consider removing this?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) instructions for Linux operation systems
+* [Windows link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/) instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux operation systems
-## 3. Deploy application
+5. How to install Docker
+* [Docker link](https://docs.docker.com/engine/install/) instructions for Windows and Linux operation systems
-To deploy application run following steps:
+6. How to install GCloud
+* [Gcloud link](https://cloud.google.com/sdk/docs/install#linux) instructions for Windows and Linux operation systems
-* Authinticate in Artifact registry
+7. How to install Terraform
+* [Terraform link](https://www.terraform.io/downloads)
Review Comment:
May we just list the required technologies as a simple pre-requisite list with links for further information? An example is:
```markdown
- [docker](https://docs.docker.com)
- [gcloud](https://cloud.google.com/sdk/docs/install)
- [terraform](https://www.terraform.io/downloads)
```
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) instructions for Linux operation systems
+* [Windows link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/) instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux operation systems
-## 3. Deploy application
+5. How to install Docker
+* [Docker link](https://docs.docker.com/engine/install/) instructions for Windows and Linux operation systems
-To deploy application run following steps:
+6. How to install GCloud
+* [Gcloud link](https://cloud.google.com/sdk/docs/install#linux) instructions for Windows and Linux operation systems
-* Authinticate in Artifact registry
+7. How to install Terraform
+* [Terraform link](https://www.terraform.io/downloads)
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
-```
+After installations of the required packages will be completed, you will need to download ***BEAM Playground*** from the repository. Please open the terminal and execute the following command:
-* Вeploy backend services
+`git clone 'url_to_github_project_with_beam_playground'`
Review Comment:
May we consider removing this?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) instructions for Linux operation systems
+* [Windows link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/) instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux operation systems
-## 3. Deploy application
+5. How to install Docker
+* [Docker link](https://docs.docker.com/engine/install/) instructions for Windows and Linux operation systems
-To deploy application run following steps:
+6. How to install GCloud
+* [Gcloud link](https://cloud.google.com/sdk/docs/install#linux) instructions for Windows and Linux operation systems
-* Authinticate in Artifact registry
+7. How to install Terraform
+* [Terraform link](https://www.terraform.io/downloads)
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
-```
+After installations of the required packages will be completed, you will need to download ***BEAM Playground*** from the repository. Please open the terminal and execute the following command:
-* Вeploy backend services
+`git clone 'url_to_github_project_with_beam_playground'`
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+Once downloading is completed you will find the "beam" folder on your disk (you can execute the `ls -la` command for Linux, or the `dir` command for Windows). Open that folder
+
+It is possible to configure BEAM to work with the different versions of the environment (like staging, production, etc.), you will need to create a special folder in `playground/terraform/environment` and put all configuration files inside:
+* File name - `terraform.tfvars`, items inside:
+```
+network_name = "network_name" #Enter the network name - you can choose any name for the network according to the naming policy above
+project_id = "project_id" #Enter the project ID - ID of created project
+gke_name = "playground-backend" #Set the GKE name - you can choose any name for Google Kubernetes Engine according to the naming policy above
+region = "us-east1" #Set the region - preferred region according to your needs
+pg_location = "us-east1-b" #Select the location - location should be in region you set before
+state_bucket = "bucket_name" #Name of bucket - Google Cloud bucket where BEAM Playground will put temp files, [terraform state file] (https://spacelift.io/blog/terraform-state)
+bucket_examples_name = "bucket_name-example" #Enter an example bucket name - bucket for some build-in examples for BEAM Playground
Review Comment:
Thank you for listing the required variables. However, I would like to propose that we leverage the capability of terraform plugins to recommend the variables within a developer's IDE. For example, if one is using visual studio code's terraform plugin and edit a terraform.tfvars file. The IDE plugin is smart to detect the contextual variables in a terraform module. It even shows what is required and optional. The value of this is that we can keep the documentation and the infrastructure-as-code orthogonal. What I mean by orthogonal is that if we have to list all the required variables here in the README, one needs to maintain the alignment of this list with the actual infrastructure-as-code. Typically as time moves along, this is not maintained.
##########
playground/terraform/infrastructure/appengine/main.tf:
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+resource "google_app_engine_application" "app_playground" {
+ project = var.project_id
+ location_id = var.region == "us-central1" ? var.location_id_us : var.region
+ database_type = "CLOUD_DATASTORE_COMPATIBILITY"
+}
+
+resource "google_project_service" "firestore" {
+ project = var.project_id
+ service = "firestore.googleapis.com"
+ disable_dependent_services = true
+ depends_on = [
+ google_app_engine_application.app_playground
+ ]
+}
+
+resource "google_app_engine_flexible_app_version" "default_app" {
+ depends_on = [
+ google_app_engine_application.app_playground
+ ]
+ count = var.create_default_service ? 1 : 0
+ service = "default"
+ version_id = "mlflow-default"
+ runtime = "custom"
+ project = var.project_id
+
+ deployment {
+ container {
+ image = "gcr.io/cloudrun/hello"
Review Comment:
Why are we deploying this image?
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) instructions for Linux operation systems
+* [Windows link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/) instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux operation systems
-## 3. Deploy application
+5. How to install Docker
+* [Docker link](https://docs.docker.com/engine/install/) instructions for Windows and Linux operation systems
-To deploy application run following steps:
+6. How to install GCloud
+* [Gcloud link](https://cloud.google.com/sdk/docs/install#linux) instructions for Windows and Linux operation systems
-* Authinticate in Artifact registry
+7. How to install Terraform
+* [Terraform link](https://www.terraform.io/downloads)
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
-```
+After installations of the required packages will be completed, you will need to download ***BEAM Playground*** from the repository. Please open the terminal and execute the following command:
-* Вeploy backend services
+`git clone 'url_to_github_project_with_beam_playground'`
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+Once downloading is completed you will find the "beam" folder on your disk (you can execute the `ls -la` command for Linux, or the `dir` command for Windows). Open that folder
+
+It is possible to configure BEAM to work with the different versions of the environment (like staging, production, etc.), you will need to create a special folder in `playground/terraform/environment` and put all configuration files inside:
+* File name - `terraform.tfvars`, items inside:
+```
+network_name = "network_name" #Enter the network name - you can choose any name for the network according to the naming policy above
+project_id = "project_id" #Enter the project ID - ID of created project
+gke_name = "playground-backend" #Set the GKE name - you can choose any name for Google Kubernetes Engine according to the naming policy above
+region = "us-east1" #Set the region - preferred region according to your needs
+pg_location = "us-east1-b" #Select the location - location should be in region you set before
+state_bucket = "bucket_name" #Name of bucket - Google Cloud bucket where BEAM Playground will put temp files, [terraform state file] (https://spacelift.io/blog/terraform-state)
+bucket_examples_name = "bucket_name-example" #Enter an example bucket name - bucket for some build-in examples for BEAM Playground
+```
+* File name - `state.tfbackend`, item inside:
```
+bucket = "bucket_name" #input bucket name - will be used for terraform tfstate file
+```
+Then, let's configure authentication for the Google Cloud Platform:
-* Deploy frontend service
+* The following command allows us to authenticate using JSON key file
+```
+ export GOOGLE_APPLICATION_CREDENTIALS=`your service account JSON key location` (absolute path)
+```
+* Using the following command, we will activate the newly created service account:
+```
+ gcloud auth activate-service-account `full principal service account` --key-file=`your service account JSON key location` (absolute path)
+```
-```bash
-./gradlew playground:terraform:deployFrontend -Pproject_environment="env-name" -Pdocker-tag="tag" ```
+# Infrastructure deployment:
+* To deploy the Infrastructure, use the following command (please be sure that you are in the "beam" folder):
+```
+./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env" (env - folder name which you created for configuration files)
+```
+# Backend deployment:
+Once the script was executed successfully, you will need to authenticate on Docker and Google Kubernetes Engine
+The following command will authorize us in the Docker registry
+```
+ cat `your service account json key locaton` | docker login -u _json_key --password-stdin https://`chosen_region`-docker.pkg.dev
Review Comment:
Since we are using Artifact Registry, wouldn't the command instead be:
```
gcloud auth configure-docker \
<REGION>-docker.pkg.dev
```
For example, if the region of the Artifact Registry repository is `us-central1`, the command would be:
```
gcloud auth configure-docker \
us-central1-docker.pkg.dev
```
##########
playground/terraform/build.gradle.kts:
##########
@@ -395,3 +372,111 @@ task("deployBackend") {
dependsOn(deploy)
}
+task("takeConfig") {
+ group = "deploy"
+ doLast {
+ var ipaddr = ""
+ var redis = ""
+ var proj = ""
+ var registry = ""
+ var ipaddrname = ""
+ var d_tag = ""
+ var stdout = ByteArrayOutputStream()
+ if (project.hasProperty("docker-tag")) {
+ d_tag = project.property("docker-tag") as String
+ }
+ exec {
+ commandLine = listOf("terraform", "output", "playground_static_ip_address")
+ standardOutput = stdout
+ }
+ ipaddr = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+
+ exec {
+ commandLine = listOf("terraform", "output", "playground_redis_ip")
+ standardOutput = stdout
+ }
+ redis = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "playground_gke_project")
+ standardOutput = stdout
+ }
+ proj = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "docker-repository-root")
+ standardOutput = stdout
+ }
+ registry = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+ exec {
+ commandLine = listOf("terraform", "output", "playground_static_ip_address_name")
+ standardOutput = stdout
+ }
+ ipaddrname = stdout.toString().trim().replace("\"", "")
+ stdout = ByteArrayOutputStream()
+
+ val configFileName = "values.yaml"
+ val modulePath = project(":playground").projectDir.absolutePath
+ val file = File("$modulePath/infrastructure/helm-backend/$configFileName")
+ val lines = file.readLines()
+ val endOfSlice = lines.indexOfFirst { it.contains("static_ip") }
+ if (endOfSlice != -1) {
+ val oldContent = lines.slice(0 until endOfSlice)
+ val flagDelete = file.delete()
+ if (!flagDelete) {
+ throw kotlin.RuntimeException("Deleting file failed")
+ }
+ val sb = kotlin.text.StringBuilder()
+ val lastLine = oldContent[oldContent.size - 1]
+ oldContent.forEach {
+ if (it == lastLine) {
+ sb.append(it)
+ } else {
+ sb.appendLine(it)
+ }
+ }
+ file.writeText(sb.toString())
+ }
+ file.appendText("""
+static_ip: ${ipaddr}
+redis_ip: ${redis}:6379
+project_id: ${proj}
+registry: ${registry}
+static_ip_name: ${ipaddrname}
+tag: $d_tag
+ """)
+ }
+}
+helm {
+ val backend by charts.creating {
+ chartName.set("backend")
+ sourceDir.set(file("../infrastructure/helm-backend"))
+ }
+ releases {
+ create("backend") {
+ from(backend)
+ }
+ }
+}
+task ("gkebackend") {
+ group = "deploy"
+ val init = tasks.getByName("terraformInit")
+ val apply = tasks.getByName("terraformApplyInf")
+ val takeConfig = tasks.getByName("takeConfig")
+ val push = tasks.getByName("pushBack")
+ val helm = tasks.getByName("helmInstallBackend")
+ val prepare = tasks.getByName("prepareConfig")
+ dependsOn(init)
+ dependsOn(apply)
+ dependsOn(takeConfig)
+ dependsOn(push)
+ dependsOn(helm)
+ dependsOn(prepare)
+ apply.mustRunAfter(init)
+ takeConfig.mustRunAfter(apply)
+ push.mustRunAfter(takeConfig)
+ helm.mustRunAfter(push)
Review Comment:
Can we consider https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/gke-deploy instead of executing within a gradle context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989107707
##########
.github/workflows/playground_deploy_backend_gke.yml:
##########
@@ -0,0 +1,65 @@
+name: kuber-bkend
+on:
+ push:
+
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ K8S_NAMESPACE: playground-backend
Review Comment:
all those variables are not used in manual deployment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r989109296
##########
.github/workflows/playground_deploy_backend_gke.yml:
##########
@@ -0,0 +1,65 @@
+name: kuber-bkend
+on:
+ push:
Review Comment:
Completed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] olehborysevych commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
olehborysevych commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r981084865
##########
.github/workflows/playground_deploy_backend_gke.yml:
##########
@@ -0,0 +1,65 @@
+name: kuber-bkend
+on:
+ push:
+
+env:
+ BEAM_ROOT_DIR: ../../
+ BEAM_EXAMPLE_CATEGORIES: ../categories.yaml
+ BEAM_VERSION: 2.40.0
+ K8S_NAMESPACE: playground-backend
Review Comment:
why this is still hardcoded here? this should depend on environment i guess
##########
playground/infrastructure/helm-backend/values.yaml:
##########
@@ -0,0 +1,36 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+replicaCount: 1
+image:
+ java_image: beam_playground-backend-java
+ go_image: beam_playground-backend-go
+ router_image: beam_playground-backend-router
+ scio_image: beam_playground-backend-scio
+ python_image: beam_playground-backend-python
+ pullPolicy: Always
+ tag: "beta"
Review Comment:
@MakarkinSAkvelon this is somewhat unclear. I know this is how it is done now but still...
##########
playground/terraform/infrastructure/gke/variables.tf:
##########
@@ -42,7 +42,6 @@ variable "name" {
variable "location" {
description = "Location of GKE cluster"
- default = "us-central1-a"
Review Comment:
@MakarkinSAkvelon also please note that location is still hardcoded for container registry. It's not present in PR but i think it's a bad idea to potentially have registry in different location
##########
playground/terraform/environment/beta/state.tfbackend:
##########
@@ -17,4 +17,4 @@
# under the License.
#
-bucket = "beam_playground_terraform"
+bucket = "pg-second"
Review Comment:
@MakarkinSAkvelon is this needed at all?
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
+* Service account with the following roles should be created:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
+* Following APIs should be enabled:
+ - Identity and Access Management (IAM)
+ - Compute Engine API
+ - App Engine Admin API
+ - Cloud Resource Manager API
+
+* Make necessary changes in `playground/terraform/environment/beta/terraform.tfvars` file:
```
-
-## 1. Create new environment
-
-To provide information about the terraform backend, run the following commands
-
-* New environment folder
-
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+network_name = "network_name" #Choose network name
+project_id = "project_id" #Input project ID
+gke_name = "playground-backend" #Define GKE name
+region = "us-east1" #Choose region
+pg_location = "us-east1-b" #Choose location (should be in the region)
+state_bucket = "bucket_name" #Input bucket name
+bucket_examples_name = "bucket_name-example" #Input example bucket name
```
-
-* Backend config
-
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
+* Make necessary changes in `playground/terraform/environment/beta/state.tfbackend` file:
```
-
-* Terraform variables config and provide necessary variables
-
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
+bucket = "bucket_name" #input bucket name (will be used for tfstate file)
```
-
-Then provide necessary variables.
-
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
+* Export GOOGLE_APPLICATION_CREDENTIALS using following command:
```
-
-## 3. Deploy application
-
-To deploy application run following steps:
-
-* Authinticate in Artifact registry
-
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+ export GOOGLE_APPLICATION_CREDENTIALS=`your json key locaton`
```
-
-* Вeploy backend services
-
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+* Activate created service account using following command:
+```
+ gcloud auth activate-service-account `full principal service account` --key-file=`your json key locaton`
Review Comment:
we need service account name here? actually if we are using json key the service account name could be omited
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
+* Service account with the following roles should be created:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
+* Following APIs should be enabled:
+ - Identity and Access Management (IAM)
+ - Compute Engine API
+ - App Engine Admin API
+ - Cloud Resource Manager API
+
+* Make necessary changes in `playground/terraform/environment/beta/terraform.tfvars` file:
```
-
-## 1. Create new environment
-
-To provide information about the terraform backend, run the following commands
-
-* New environment folder
-
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+network_name = "network_name" #Choose network name
+project_id = "project_id" #Input project ID
+gke_name = "playground-backend" #Define GKE name
+region = "us-east1" #Choose region
+pg_location = "us-east1-b" #Choose location (should be in the region)
+state_bucket = "bucket_name" #Input bucket name
+bucket_examples_name = "bucket_name-example" #Input example bucket name
```
-
-* Backend config
-
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
+* Make necessary changes in `playground/terraform/environment/beta/state.tfbackend` file:
```
-
-* Terraform variables config and provide necessary variables
-
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
+bucket = "bucket_name" #input bucket name (will be used for tfstate file)
```
-
-Then provide necessary variables.
-
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
+* Export GOOGLE_APPLICATION_CREDENTIALS using following command:
```
-
-## 3. Deploy application
-
-To deploy application run following steps:
-
-* Authinticate in Artifact registry
-
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+ export GOOGLE_APPLICATION_CREDENTIALS=`your json key locaton`
```
-
-* Вeploy backend services
-
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+* Activate created service account using following command:
+```
+ gcloud auth activate-service-account `full principal service account` --key-file=`your json key locaton`
+```
+* Install kubectl:
+```
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+```
+* Install Helm:
+```
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+```
+# Infrastructure deployment:
+* Run following command for infrastructure deployment (please be sure that you are in the "beam" folder):
+```
+./gradlew playground:terraform:InitInfrastructure -Pproject_environment="beta"
+```
+# Backend deployment:
+* Login to Docker registry:
+```
+ cat `your json key locaton` | docker login -u _json_key --password-stdin https://`chosen_region`-docker.pkg.dev
+```
+* Login to GKE
+```
+gcloud container clusters get-credentials --region `chosen_pg_location` `gke_name` --project `project_id`
+```
+* Database index creation:
+```
+gcloud app deploy playground/index.yaml --project=`project_id`
+```
+* Run following command for backend deployment (please be sure that you are in the "beam" folder):
+```
+./gradlew playground:terraform:gkebackend -Pproject_environment="beta" -Pdocker-tag="beta"
Review Comment:
do we need both parameters? Does it make sense to pass different values here?
##########
.github/workflows/playground_deploy_backend_gke.yml:
##########
@@ -0,0 +1,65 @@
+name: kuber-bkend
+on:
+ push:
Review Comment:
@MakarkinSAkvelon maybe we need to switch to manual event?
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
+* Service account with the following roles should be created:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
+* Following APIs should be enabled:
+ - Identity and Access Management (IAM)
+ - Compute Engine API
+ - App Engine Admin API
+ - Cloud Resource Manager API
+
+* Make necessary changes in `playground/terraform/environment/beta/terraform.tfvars` file:
```
-
-## 1. Create new environment
-
-To provide information about the terraform backend, run the following commands
-
-* New environment folder
-
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+network_name = "network_name" #Choose network name
+project_id = "project_id" #Input project ID
+gke_name = "playground-backend" #Define GKE name
+region = "us-east1" #Choose region
+pg_location = "us-east1-b" #Choose location (should be in the region)
+state_bucket = "bucket_name" #Input bucket name
+bucket_examples_name = "bucket_name-example" #Input example bucket name
```
-
-* Backend config
-
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
+* Make necessary changes in `playground/terraform/environment/beta/state.tfbackend` file:
```
-
-* Terraform variables config and provide necessary variables
-
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
+bucket = "bucket_name" #input bucket name (will be used for tfstate file)
Review Comment:
why do we have to provide this bucket name twice?
##########
playground/terraform/infrastructure/appengine/main.tf:
##########
@@ -0,0 +1,52 @@
+resource "google_app_engine_application" "app_playground" {
+ project = var.project_id
+ location_id = var.region
+ database_type = "CLOUD_DATASTORE_COMPATIBILITY"
+}
+
+resource "google_project_service" "firestore" {
+ project = var.project_id
+ service = "firestore.googleapis.com"
+ disable_dependent_services = true
+ depends_on = [
+ google_app_engine_application.app_playground
+ ]
+}
+
+resource "google_app_engine_flexible_app_version" "default_app" {
Review Comment:
@MakarkinSAkvelon not sure we need this
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
Review Comment:
Please add some info regarding what is the purpose of this bucket
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
+* Service account with the following roles should be created:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
+* Following APIs should be enabled:
+ - Identity and Access Management (IAM)
+ - Compute Engine API
+ - App Engine Admin API
+ - Cloud Resource Manager API
+
+* Make necessary changes in `playground/terraform/environment/beta/terraform.tfvars` file:
```
-
-## 1. Create new environment
-
-To provide information about the terraform backend, run the following commands
-
-* New environment folder
-
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+network_name = "network_name" #Choose network name
+project_id = "project_id" #Input project ID
+gke_name = "playground-backend" #Define GKE name
+region = "us-east1" #Choose region
+pg_location = "us-east1-b" #Choose location (should be in the region)
+state_bucket = "bucket_name" #Input bucket name
+bucket_examples_name = "bucket_name-example" #Input example bucket name
```
-
-* Backend config
-
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
+* Make necessary changes in `playground/terraform/environment/beta/state.tfbackend` file:
```
-
-* Terraform variables config and provide necessary variables
-
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
+bucket = "bucket_name" #input bucket name (will be used for tfstate file)
```
-
-Then provide necessary variables.
-
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
+* Export GOOGLE_APPLICATION_CREDENTIALS using following command:
```
-
-## 3. Deploy application
-
-To deploy application run following steps:
-
-* Authinticate in Artifact registry
-
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+ export GOOGLE_APPLICATION_CREDENTIALS=`your json key locaton`
Review Comment:
need to add some link to how can we obtain credentials and for which account
Also please add if this should be a absolute or relative path
##########
playground/infrastructure/helm-backend/values.yaml:
##########
@@ -0,0 +1,36 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+replicaCount: 1
+image:
+ java_image: beam_playground-backend-java
+ go_image: beam_playground-backend-go
+ router_image: beam_playground-backend-router
+ scio_image: beam_playground-backend-scio
+ python_image: beam_playground-backend-python
+ pullPolicy: Always
+ tag: "beta"
+
+service:
+ type: NodePort
+ targetPort: 8080
+ port: 443
+
+autoscaling:
+ enabled: false
Review Comment:
@MakarkinSAkvelon why aren't we enabling scaling?
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
+* Service account with the following roles should be created:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
+* Following APIs should be enabled:
+ - Identity and Access Management (IAM)
+ - Compute Engine API
+ - App Engine Admin API
+ - Cloud Resource Manager API
+
+* Make necessary changes in `playground/terraform/environment/beta/terraform.tfvars` file:
```
-
-## 1. Create new environment
-
-To provide information about the terraform backend, run the following commands
-
-* New environment folder
-
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+network_name = "network_name" #Choose network name
+project_id = "project_id" #Input project ID
+gke_name = "playground-backend" #Define GKE name
+region = "us-east1" #Choose region
+pg_location = "us-east1-b" #Choose location (should be in the region)
+state_bucket = "bucket_name" #Input bucket name
+bucket_examples_name = "bucket_name-example" #Input example bucket name
```
-
-* Backend config
-
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
+* Make necessary changes in `playground/terraform/environment/beta/state.tfbackend` file:
```
-
-* Terraform variables config and provide necessary variables
-
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
+bucket = "bucket_name" #input bucket name (will be used for tfstate file)
```
-
-Then provide necessary variables.
-
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
+* Export GOOGLE_APPLICATION_CREDENTIALS using following command:
```
-
-## 3. Deploy application
-
-To deploy application run following steps:
-
-* Authinticate in Artifact registry
-
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+ export GOOGLE_APPLICATION_CREDENTIALS=`your json key locaton`
```
-
-* Вeploy backend services
-
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+* Activate created service account using following command:
+```
+ gcloud auth activate-service-account `full principal service account` --key-file=`your json key locaton`
+```
+* Install kubectl:
+```
+ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" &&\
+ chmod +x kubectl &&\
+ mv kubectl /usr/local/bin/
+```
+* Install Helm:
+```
+ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 &&\
+ chmod 700 get_helm.sh &&\
+ ./get_helm.sh
+```
+# Infrastructure deployment:
+* Run following command for infrastructure deployment (please be sure that you are in the "beam" folder):
+```
+./gradlew playground:terraform:InitInfrastructure -Pproject_environment="beta"
+```
+# Backend deployment:
+* Login to Docker registry:
+```
+ cat `your json key locaton` | docker login -u _json_key --password-stdin https://`chosen_region`-docker.pkg.dev
Review Comment:
For me "gcloud auth configure-docker" was needed prior to this step
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
+* Service account with the following roles should be created:
Review Comment:
Looks like we also need appengine.deployer role
##########
playground/terraform/environment/beta/state.tfbackend:
##########
@@ -17,4 +17,4 @@
# under the License.
#
-bucket = "beam_playground_terraform"
+bucket = "pg-second"
Review Comment:
@MakarkinSAkvelon can we have a meaningful name here?
##########
playground/terraform/README.md:
##########
@@ -17,73 +17,89 @@
under the License.
-->
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Prerequisit:
+
+* GCP project should be created
+* Bucket should be created
Review Comment:
If this is a bucket for terraform state - could you please describe that local terraform backend can be used if applicable. Also we should at least point to some doc describing best settings for bucket (versioning, region...)
##########
playground/terraform/variables.tf:
##########
@@ -85,10 +85,9 @@ variable "repository_id" {
default = "playground-repository"
}
-variable "repository_location" {
- description = "Location of Artifact Registry"
- default = "us-central1"
-}
+# variable "repository_location" {
Review Comment:
@MakarkinSAkvelon please remove comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] MakarkinSAkvelon commented on a diff in pull request #23208: [Playground] Helm+kubernetes+deployment
Posted by GitBox <gi...@apache.org>.
MakarkinSAkvelon commented on code in PR #23208:
URL: https://github.com/apache/beam/pull/23208#discussion_r993697984
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,179 @@
specific language governing permissions and limitations
under the License.
-->
+# Here you will find the steps for deploying BEAM Playground on your project
-# Requirements
+## Prerequisites:
-The following items need to be setup for the Playground cluster deployment on GCP:
+### Before you will start with the installation steps, please read this carefully:
+1. How to create a new project in Google Cloud Platform:
+```
+https://cloud.google.com/resource-manager/docs/creating-managing-projects
+```
+**Description:** * *How to create a new project in Google Cloud Platform. It's strongly recommended to use the new Google Cloud Project for your new projects* *
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
+2. How to create a new service account
+```
+https://cloud.google.com/iam/docs/creating-managing-service-accounts
+```
+**Description:** * *You will find the instruction "How to create Service account" for your new project* *
-# Deployment steps
+3. How to create a JSON key for a Google Service account:
+```
+https://cloud.google.com/iam/docs/creating-managing-service-account-keys
+```
+**Description:** * *In this instruction will be explained how to create a JSON key for your service account. it will be required for authentication.* *
-## 0. Create GCS bucket for state
+4. How to create a new bucket:
+```
+https://cloud.google.com/storage/docs/creating-buckets
+```
+**Description:** * *Short instruction for bucket creation. You will need at least one bucket for file keeping* *
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+5. How to assign a new role for the service account:
+```
+https://cloud.google.com/iam/docs/granting-changing-revoking-access
```
+**Description:** * *It will explain how to assign required roles for your service account* *
-## 1. Create new environment
+6. How to install gcloud CLI to your local environment
+```
+https://cloud.google.com/sdk/docs/install?hl=en
+```
+**Description:** * *GCloud - Google command line interface, which will allow you to interact with the Google cloud platform thru the command line (https://en.wikipedia.org/wiki/Command-line_interface)* *
-To provide information about the terraform backend, run the following commands
+7. What are Google Cloud Platform APIs
+```
+https://cloud.google.com/apis
+```
+**Description:** * *Short description of all Google Platform APIs* *
-* New environment folder
+8. How to Enable Google Cloud Platform API
+```
+https://cloud.google.com/endpoints/docs/openapi/enable-api
+```
+**Description:** * *Steps to enable APIs in Google Cloud Platform* *
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+9. Google Cloud Platform naming policy
+```
+https://cloud.google.com/compute/docs/naming-resources
+```
+**Description:** * *Describes the naming convention for Compute Engine resources.* *
+10. What is Docker
```
+https://docs.docker.com/get-started/overview/
+```
+**Description:** * *What is Docker and how to use it* *
-* Backend config
+***Google Cloud preparation steps:***
+After you created or identified an existing project for deployment, add or select a service account with a JSON key, and add or select state_bucket for storing the Terraform state data according to the instructions above:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
-```
+These Google APIs will be needed for the BEAM Playground
+ - **Identity and Access Management (IAM)** - Manages identity and access control for Google Cloud Platform resources, including the creation of service accounts, which you can use to authenticate to Google and make API calls
+ - **Compute Engine API** - Creates and runs Virtual Machines on Google Cloud
+ - **App Engine Admin API** - Provisions and manages App Engine applications
+ - **Cloud Resource Manager API** - Creates, reads, and updates metadata for Google Cloud Platform resource containers
+You may find the instruction "How to enable Google Cloud Platform API" above
-* Terraform variables config and provide necessary variables
+***Operation System preparation steps:***
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
-```
+During the Operation system preparation steps, you will need to install Java, HELM, GIT, Docker, GCloud, Terraform, Kubernetes command line interface
-Then provide necessary variables.
+1. How to install Java:
+* [This link](https://adoptopenjdk.net/) will provide you will all necessary instructions for Windows and Linux operation systems
-## 2. Provision infrastructure
+2. How to install Kubernetes Command Line Interface
+* [Linux link](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) instructions for Linux operation systems
+* [Windows link](https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/) instructions for Windows operation systems
-To deploy Playground infrastructure run gradle task:
+3. How to install HELM
+* [HELM link](https://helm.sh/docs/intro/install/) instructions for Windows and Linux operation systems
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
-```
+4. How to install GIT
+* [Git link](https://git-scm.com/download/win) instructions for Windows operation systems
+* [Git Link](https://git-scm.com/download/linux) instructions for Linux operation systems
-## 3. Deploy application
+5. How to install Docker
+* [Docker link](https://docs.docker.com/engine/install/) instructions for Windows and Linux operation systems
-To deploy application run following steps:
+6. How to install GCloud
+* [Gcloud link](https://cloud.google.com/sdk/docs/install#linux) instructions for Windows and Linux operation systems
-* Authinticate in Artifact registry
+7. How to install Terraform
+* [Terraform link](https://www.terraform.io/downloads)
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
-```
+After installations of the required packages will be completed, you will need to download ***BEAM Playground*** from the repository. Please open the terminal and execute the following command:
-* Вeploy backend services
+`git clone 'url_to_github_project_with_beam_playground'`
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org