You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/11/11 00:09:01 UTC
[GitHub] [beam] damondouglas commented on a diff in pull request #23928: [Playground] Move Playground in GKE and Infrastructure change
damondouglas commented on code in PR #23928:
URL: https://github.com/apache/beam/pull/23928#discussion_r1019718950
##########
playground/terraform/README.md:
##########
@@ -16,74 +16,117 @@
specific language governing permissions and limitations
under the License.
-->
-
-# Requirements
-
-The following items need to be setup for the Playground cluster deployment on GCP:
-
-* [GCP account](https://cloud.google.com/)
-* [`gcloud` command-line tool](https://cloud.google.com/sdk/gcloud) and required setup i.e. login
-* [Terraform](https://www.terraform.io/downloads.html) tool
-* [Docker](https://www.docker.com/get-started)
-
-# Deployment steps
-
-## 0. Create GCS bucket for state
-
-```bash
-$ gsutil mb -p ${PROJECT_ID} gs://state-bucket-name
-$ gsutil versioning set on gs://state-bucket-name
+# Playground deployment on GCP
+This guide shows you how to deploy full Playground environment on Google Cloud Platform (GCP) environment.
+
+## Prerequisites:
+
+### Following items need to be setup for Playground deployment on GCP:
+1. [GCP project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
+
+2. [GCP User account](https://cloud.google.com/appengine/docs/standard/access-control?tab=python) _(Note: You will find the instruction "How to create User account" for your new project)_<br>
+Ensure that the account has at least following privileges:
+ - App Engine Admin
+ - App Engine Creator
+ - Artifact Registry Administrator
+ - Cloud Memorystore Redis Admin
+ - Compute Admin
+ - Create Service Accounts
+ - Kubernetes Engine Admin
+ - Quota Administrator
+ - Role Administrator
+ - Security Admin
+ - Service Account User
+ - Storage Admin
+ - Cloud Datastore Index Admin
+
+3. [Google Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for saving deployment state
+
+4. DNS name for your Playground deployment instance
+
+5. OS with installed software listed below:
+
+* [Java](https://adoptopenjdk.net/)
+* [Kubernetes Command Line Interface](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)
+* [HELM](https://helm.sh/docs/intro/install/)
+* [Docker](https://docs.docker.com/engine/install/)
+* [Terraform](https://www.terraform.io/downloads)
+* [gcloud CLI](https://cloud.google.com/sdk/docs/install-sdk)
+
+6. Apache Beam Git repository cloned locally
+
+# Prepare deployment configuration:
+Playground uses `terraform.tfvars` located in `playground/terraform/environment/environment_name` to define variables specific to an environment (e.g., prod, test, staging).<br>
+1. Create a folder (further referred as `environment_name`) to define a new environment and place configuration files into it:
+
+* `terraform.tfvars` environment variables:
```
-
-## 1. Create new environment
-
-To provide information about the terraform backend, run the following commands
-
-* New environment folder
-
-```bash
-mkdir /path/to/beam/playground/terraform/environment/{env-name}
+project_id = "project_id" #GCP Project ID
+network_name = "network_name" #GCP VPC Network Name for Playground deployment
+gke_name = "playground-backend" #Playground GKE Cluster name
+region = "us-east1" #Set the deployment region
+location = "us-east1-b" #Select the deployment location from available in the specified region
+state_bucket = "bucket_name" #GCS bucket name for Beam Playground temp files
+bucket_examples_name = "bucket_name-example" #GCS bucket name for Playground examples storage
```
-
-* Backend config
-
-```bash
-echo 'bucket = "put your state bucket name here"' > /path/to/beam/playground/terraform/environment/{env-name}/state.tfbackend
+* `state.tfbackend` environment variables:
```
-
-* Terraform variables config and provide necessary variables
-
-```bash
-touch /path/to/beam/playground/terraform/environment/{env-name}/terraform.tfvars
+bucket = "bucket_name" #input bucket name - will be used for terraform tfstate file
```
-
-Then provide necessary variables.
-
-## 2. Provision infrastructure
-
-To deploy Playground infrastructure run gradle task:
-
-```bash
-./gradlew playground:terraform:InitInfrastructure -Pproject_environment="env-name"
+2. Configure authentication for the Google Cloud Platform
```
-
-## 3. Deploy application
-
-To deploy application run following steps:
-
-* Authinticate in Artifact registry
-
-```bash
-gcloud auth configure-docker us-central1-docker.pkg.dev
+gcloud init
+```
+```
+gcloud auth application-default login
```
+# Deploy Playground infrastructure:
+1. Start the following command from the top level repository folder ("beam") to deploy the Payground infrastructure:
+```
+./gradlew playground:terraform:InitInfrastructure -Pproject_environment="environment_name" -Pdns-name="playground.zone"
+```
+Where playground.zone - chosen DNS for Playground
-* Š’eploy backend services
+2. Find a Static IP in your GCP project>VPC Network>IP Addresses>pg-static-ip
+<br>Add following DNS A records for the discovered static IP address:
+```
+java.playground.zone
+python.playground.zone
+scio.playground.zone
+go.playground.zone
+router.playground.zone
+playground.zone
+```
+Where "playground.zone" is the registered DNS zone<br>
+[More about DNS zone registration](https://domains.google/get-started/domain-search/)<br>
+[More about A records in DNS](https://support.google.com/a/answer/2579934?hl=en)
-```bash
-./gradlew playground:terraform:deployBackend -Pproject_environment="env-name" -Pdocker-tag="tag"
+# Deploy Playground to Kubernetes:
+1. Run the following command to authenticate in the Docker registry:
```
+ gcloud auth configure-docker `chosen_region`-docker.pkg.dev
+```
+2. Run the following command to authenticate in GKE:
+```
+gcloud container clusters get-credentials --region `chosen_location` `gke_name` --project `project_id`
+```
+Start the following command from the top level repository folder ("beam") to deploy the Payground infrastructure:
+```
+./gradlew playground:terraform:gkebackend -Pproject_environment="environment_name" -Pdocker-tag="tag" -Pdns-name="playground.zone"
Review Comment:
When I ran this command, I received the error:
```
> Task :playground:frontend:docker FAILED
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':playground:terraform:helmInstallPlaygroundToDefault'.
> There was a failure while executing work items
> A failure occurred while executing org.unbrokendome.gradle.plugins.helm.command.HelmExecWorkAction
> A problem occurred starting process 'command 'helm''
```
I run helm version, I see:
```
version.BuildInfo{Version:"v3.9.4", GitCommit:"dbc6d8e20fe1d58d50e6ed30f09a04a77e4c68db", GitTreeState:"clean", GoVersion:"go1.17.13"}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org