You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "damondouglas (via GitHub)" <gi...@apache.org> on 2023/03/30 22:24:39 UTC
[GitHub] [beam] damondouglas commented on a diff in pull request #25793: Automation: Tour of Beam infrastructure deployment
damondouglas commented on code in PR #25793:
URL: https://github.com/apache/beam/pull/25793#discussion_r1153815229
##########
learning/tour-of-beam/terraform/README.md:
##########
@@ -0,0 +1,121 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+# The Tour of Beam deployment on GCP
+This guide provides instructions on how to deploy the Tour of Beam environment on Google Cloud Platform (GCP) and Firebase environment. Before starting the deployment, ensure that you have the following prerequisites in place:
+
+## Prerequisites:
+
+1. [GCP project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
+2. [GCP User account](https://cloud.google.com/appengine/docs/standard/access-control?tab=python) _(Note: You will find the instruction "How to create User account" for your new project)_<br>
+ Ensure that the account has at least following privileges:
+ - Cloud Datastore Owner
+ - Create Service Accounts
+ - Security Admin
+ - Service Account User
+ - Service Usage Admin
+ - Storage Admin
+ - Kubernetes Engine Cluster Viewer
+
+3. [Google Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for saving deployment state
+
+4. An OS with the following software installed:
+
+* [Java](https://adoptopenjdk.net/)
+* [NodeJS & npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm/)
+* [Flutter (3.7.3 >)](https://docs.flutter.dev/get-started/install)
+* [Dart SDK (2.19.2)](https://dart.dev/get-dart)
+* [Firebase-tools CLI](https://www.npmjs.com/package/firebase-tools)
+* [Terraform](https://www.terraform.io/downloads)
+* [gcloud CLI](https://cloud.google.com/sdk/docs/install-sdk)
+* [Kubectl authentication plugin](https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke)
+* [Go](https://go.dev/doc/install)
+
+5. Apache Beam Git repository cloned locally
+
+# Prepare deployment configuration:
+Tour of Beam backend uses `terraform.tfvars` located in `learning/tour-of-beam/terraform/environment/environment_name/` to define variables specific to an environment (e.g., prod, test, staging). Follow the steps below to prepare the deployment configuration:<br>
+1. Create a folder (referred to as `environment_name`) to define a new environment and place configuration files into it:
+
+* `terraform.tfvars` environment variables:
+```
+project_id = "gcp_project_id" # Your GCP Project ID
+cloudfunctions_bucket = "gcs_bucket_name" # Globally unique name of the bucket to store cloud functions' source code
+region = "gcp_region" # Your GCP resources region
+service_account_id = "service_account_name" # Name of GCP service account to run Tour of Beam cloud functions
+
+```
+* `state.tfbackend` environment variables:
+```
+bucket = "bucket_name" # Your created bucket name for terraform tfstate file
+```
+2. Configure authentication for the Google Cloud Platform (GCP)
+```
+gcloud init
+```
+```
+gcloud auth application-default login
+```
+
+3. Configure authentication in the GCP Docker registry:
+```
+ gcloud auth configure-docker `chosen_region`-docker.pkg.dev
+```
+4. And the authentication in GCP Google Kubernetes Engine:
+```
+gcloud container clusters get-credentials --region `chosen_gke_zone` `gke_name` --project `project_id`
Review Comment:
This step requires an existing cluster but I didn't see this as a pre-requisite.
##########
learning/tour-of-beam/terraform/build.gradle.kts:
##########
@@ -0,0 +1,388 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.pswidersk.gradle.terraform.TerraformTask
+import java.io.ByteArrayOutputStream
+import java.util.regex.Pattern
+
+plugins {
+ id("com.pswidersk.terraform-plugin") version "1.0.0"
+}
+
+terraformPlugin {
+ terraformVersion.set("1.0.9")
+}
+
+/* init Infrastructure for migrate */
+tasks.register<TerraformTask>("terraformInit") {
+ // exec args can be passed by commandline, for example
+ var environment = project.property("project_environment") as String
+ args(
+ "init", "-migrate-state",
+ "-backend-config=./environment/$environment/state.tfbackend",
+ "-var=environment=$environment",
+ if (file("./environment/$environment/terraform.tfvars").exists()) {
+ "-var-file=./environment/$environment/terraform.tfvars"
+ } else {
+ "-no-color"
+ }
+ )
+ }
+
+ /* refresh Infrastucture for remote state */
+tasks.register<TerraformTask>("terraformRef") {
+ var environment = project.property("project_environment") as String
+ args(
+ "refresh",
+ "-lock=false",
+ "-var=environment=$environment",
+ if (file("./environment/$environment/terraform.tfvars").exists()) {
+ "-var-file=./environment/$environment/terraform.tfvars"
+ } else {
+ "-no-color"
+ }
+ )
+ }
+
+tasks.register<TerraformTask>("terraformApplyBackend") {
+ group = "backend-deploy"
+ var pg_router_host = project.extensions.extraProperties["pg_router_host"] as String
+ var environment = project.property("project_environment") as String
+ var gcloud_account = project.property("gcloud_account") as String
+ args(
+ "apply",
+ "-auto-approve",
+ "-lock=false",
+ "-parallelism=3",
+ "-var=pg_router_host=$pg_router_host",
+ "-var=gcloud_init_account=$gcloud_account",
+ "-var=environment=$environment",
+ if (file("./environment/$environment/terraform.tfvars").exists()) {
+ "-var-file=./environment/$environment/terraform.tfvars"
+ } else {
+ "-no-color"
+ }
+ )
+ tasks.getByName("uploadLearningMaterials").mustRunAfter(this)
+
+ }
+
+tasks.register<TerraformTask>("terraformDestroy") {
+ var pg_router_host = project.extensions.extraProperties["pg_router_host"] as String
+ var environment = project.property("project_environment") as String
+ var gcloud_account = project.property("gcloud_account") as String
+ args(
+ "destroy",
+ "-auto-approve",
+ "-lock=false",
+ "-var=pg_router_host=$pg_router_host",
+ "-var=environment=$environment",
+ "-var=gcloud_init_account=$gcloud_account",
+ if (file("./environment/$environment/terraform.tfvars").exists()) {
+ "-var-file=./environment/$environment/terraform.tfvars"
+ } else {
+ "-no-color"
+ }
+ )
+}
+
+tasks.register("getRouterHost") {
+ group = "backend-deploy"
+ val result = ByteArrayOutputStream()
+ exec {
+ commandLine("kubectl", "get", "svc", "-l", "app=backend-router-grpc", "-o", "jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}:{.items[0].spec.ports[0].port}'")
+ standardOutput = result
+ }
+ val pg_router_host = result.toString().trim().replace("'", "")
+ project.extensions.extraProperties["pg_router_host"] = pg_router_host
+}
+
+tasks.register("indexcreate") {
+ group = "backend-deploy"
+ val indexpath = "../backend/internal/storage/index.yaml"
+ exec {
+ executable("gcloud")
+ args("datastore", "indexes", "create", indexpath)
+ }
+}
+
+tasks.register("firebaseProjectCreate") {
+ group = "frontend-deploy"
+ val result = ByteArrayOutputStream()
+ var project_id = project.property("project_id") as String
+ exec {
+ executable("firebase")
+ args("projects:list")
+ standardOutput = result
+ }
+ val output = result.toString().trim()
+ if (output.contains(project_id)) {
+ println("Firebase is already added to project $project_id.")
+ } else {
+ exec {
+ executable("firebase")
+ args("projects:addfirebase", project_id)
+ }.assertNormalExitValue()
+ println("Firebase has been added to project $project_id.")
+ }
+}
Review Comment:
Why can't someone just use the firebase cli directly?
##########
learning/tour-of-beam/terraform/cloud_functions/variables.tf:
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+variable "service_account_id" {
+ description = "The name of Service Account to run Cloud Function"
+}
+
+variable "project_id" {
+ description = "The GCP Project ID of function"
+}
+
+variable "region" {
+ description = "The GCP Region of function"
+}
+
+variable "source_archive_bucket" {
+ description = "The GCS bucket containing the zip archive which contains the function"
+}
+variable "source_archive_object" {
+ description = "The source archive object (file) in archive bucket"
+}
+
+variable "entry_point_names" {
+ type = list
Review Comment:
Why not refactor the backend to route the get requests?
##########
learning/tour-of-beam/terraform/build.gradle.kts:
##########
@@ -0,0 +1,388 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.pswidersk.gradle.terraform.TerraformTask
+import java.io.ByteArrayOutputStream
+import java.util.regex.Pattern
+
+plugins {
+ id("com.pswidersk.terraform-plugin") version "1.0.0"
Review Comment:
I personally think the project should depend on a gradle plugin last updated 3 years ago. I suspect the reason for the plugin is to simplify all of the variables and the use of various environments. That complexity could be reduced by using a prefix model. For example, one of the variables is called `cloudfunctions_bucket`. Instead of asking for one variable that names a resource, perhaps you could do this for example:
```
variable "resource_name_prefix" {
type = string
description = "The resource name prefix applied to all resource naming for the application"
default = "tour-of-beam"
}
```
resource "random_string" "id" {
length = 6
upper = false
special = false
}
```
locals {
cloudfunctions_bucket = "${var.resource_name_prefix}-cfstorage-${random_string.id.result}"
}
```
This strategy eliminates the need to ask for many different resource names but also avoid the conflicts associated with global resource names such as a Google Cloud storage bucket.
##########
learning/tour-of-beam/terraform/README.md:
##########
@@ -0,0 +1,121 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+# The Tour of Beam deployment on GCP
+This guide provides instructions on how to deploy the Tour of Beam environment on Google Cloud Platform (GCP) and Firebase environment. Before starting the deployment, ensure that you have the following prerequisites in place:
+
+## Prerequisites:
+
+1. [GCP project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
+2. [GCP User account](https://cloud.google.com/appengine/docs/standard/access-control?tab=python) _(Note: You will find the instruction "How to create User account" for your new project)_<br>
+ Ensure that the account has at least following privileges:
+ - Cloud Datastore Owner
+ - Create Service Accounts
+ - Security Admin
+ - Service Account User
+ - Service Usage Admin
+ - Storage Admin
+ - Kubernetes Engine Cluster Viewer
+
+3. [Google Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for saving deployment state
+
+4. An OS with the following software installed:
+
+* [Java](https://adoptopenjdk.net/)
+* [NodeJS & npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm/)
+* [Flutter (3.7.3 >)](https://docs.flutter.dev/get-started/install)
+* [Dart SDK (2.19.2)](https://dart.dev/get-dart)
+* [Firebase-tools CLI](https://www.npmjs.com/package/firebase-tools)
+* [Terraform](https://www.terraform.io/downloads)
+* [gcloud CLI](https://cloud.google.com/sdk/docs/install-sdk)
+* [Kubectl authentication plugin](https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke)
+* [Go](https://go.dev/doc/install)
+
+5. Apache Beam Git repository cloned locally
+
+# Prepare deployment configuration:
+Tour of Beam backend uses `terraform.tfvars` located in `learning/tour-of-beam/terraform/environment/environment_name/` to define variables specific to an environment (e.g., prod, test, staging). Follow the steps below to prepare the deployment configuration:<br>
+1. Create a folder (referred to as `environment_name`) to define a new environment and place configuration files into it:
+
+* `terraform.tfvars` environment variables:
+```
+project_id = "gcp_project_id" # Your GCP Project ID
+cloudfunctions_bucket = "gcs_bucket_name" # Globally unique name of the bucket to store cloud functions' source code
+region = "gcp_region" # Your GCP resources region
+service_account_id = "service_account_name" # Name of GCP service account to run Tour of Beam cloud functions
+
+```
Review Comment:
Instead of putting terraform.tfvars in a separate directory, why not just have a common.tfvars in the terraform directory? The reason why its better to have the .tfvars file in the same directory is that it enables IDE terraform plugin autocompletion. Then all someone needs to do is this:
```
gcloud config set project <project>
terraform apply -var-file=common.tfvars -var=project_id=$(gcloud config get-value project)
```
In `common.tfvars` put:
```
region = "us-central1"
service_account_id = "tour-of-beam"
```
FYI, one can refer to multiple -var-file arguments, if you need to:
```
terraform apply -var-file=common.tfvars -var-file=bla/blab/environment_name/blablab.tfvars
```
##########
learning/tour-of-beam/terraform/cloud_functions/variables.tf:
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+variable "service_account_id" {
+ description = "The name of Service Account to run Cloud Function"
+}
+
+variable "project_id" {
+ description = "The GCP Project ID of function"
+}
+
+variable "region" {
+ description = "The GCP Region of function"
+}
+
+variable "source_archive_bucket" {
+ description = "The GCS bucket containing the zip archive which contains the function"
Review Comment:
See comment above about using a resource-prefix to prevent having to ask about different variables for global resources. In this example, perhaps one could do the following, continuing from the previous comment's example. Instead of a `source_archive_bucket` variable:
```
locals {
source_archive_bucket = "${var.resource_name_prefix}-archive-${random_string.id.result}"
}
```
##########
learning/tour-of-beam/terraform/README.md:
##########
@@ -0,0 +1,121 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+# The Tour of Beam deployment on GCP
+This guide provides instructions on how to deploy the Tour of Beam environment on Google Cloud Platform (GCP) and Firebase environment. Before starting the deployment, ensure that you have the following prerequisites in place:
+
+## Prerequisites:
+
+1. [GCP project](https://cloud.google.com/resource-manager/docs/creating-managing-projects)
+2. [GCP User account](https://cloud.google.com/appengine/docs/standard/access-control?tab=python) _(Note: You will find the instruction "How to create User account" for your new project)_<br>
+ Ensure that the account has at least following privileges:
+ - Cloud Datastore Owner
+ - Create Service Accounts
+ - Security Admin
+ - Service Account User
+ - Service Usage Admin
+ - Storage Admin
+ - Kubernetes Engine Cluster Viewer
+
+3. [Google Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for saving deployment state
+
+4. An OS with the following software installed:
+
+* [Java](https://adoptopenjdk.net/)
+* [NodeJS & npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm/)
Review Comment:
Is this only needed because of the firebase cli? If so, let the tooling refer to any additional requirements.
##########
learning/tour-of-beam/terraform/build.gradle.kts:
##########
@@ -0,0 +1,388 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.pswidersk.gradle.terraform.TerraformTask
+import java.io.ByteArrayOutputStream
+import java.util.regex.Pattern
+
+plugins {
+ id("com.pswidersk.terraform-plugin") version "1.0.0"
+}
+
+terraformPlugin {
+ terraformVersion.set("1.0.9")
+}
+
+/* init Infrastructure for migrate */
+tasks.register<TerraformTask>("terraformInit") {
+ // exec args can be passed by commandline, for example
+ var environment = project.property("project_environment") as String
+ args(
+ "init", "-migrate-state",
+ "-backend-config=./environment/$environment/state.tfbackend",
+ "-var=environment=$environment",
+ if (file("./environment/$environment/terraform.tfvars").exists()) {
+ "-var-file=./environment/$environment/terraform.tfvars"
+ } else {
+ "-no-color"
+ }
+ )
+ }
+
+ /* refresh Infrastucture for remote state */
+tasks.register<TerraformTask>("terraformRef") {
+ var environment = project.property("project_environment") as String
+ args(
+ "refresh",
+ "-lock=false",
+ "-var=environment=$environment",
+ if (file("./environment/$environment/terraform.tfvars").exists()) {
+ "-var-file=./environment/$environment/terraform.tfvars"
+ } else {
+ "-no-color"
+ }
+ )
+ }
+
+tasks.register<TerraformTask>("terraformApplyBackend") {
+ group = "backend-deploy"
+ var pg_router_host = project.extensions.extraProperties["pg_router_host"] as String
+ var environment = project.property("project_environment") as String
+ var gcloud_account = project.property("gcloud_account") as String
+ args(
+ "apply",
+ "-auto-approve",
Review Comment:
This is dangerous.
##########
learning/tour-of-beam/terraform/build.gradle.kts:
##########
@@ -0,0 +1,388 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.pswidersk.gradle.terraform.TerraformTask
+import java.io.ByteArrayOutputStream
+import java.util.regex.Pattern
+
+plugins {
+ id("com.pswidersk.terraform-plugin") version "1.0.0"
+}
+
+terraformPlugin {
+ terraformVersion.set("1.0.9")
Review Comment:
What is the highest terraform version, this plugin will support? What if the Google Cloud terraform provider's terraform version exceeds this plugin's version? The alleged benefit of using the plugin to simplify terraform commands comes at a risk that in my opinion is not worth it.
##########
learning/tour-of-beam/terraform/cloud_functions/main.tf:
##########
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+resource "google_cloudfunctions_function" "cloud_function" {
+ count = length(var.entry_point_names)
+ name = "${var.environment}_${var.entry_point_names[count.index]}"
+ runtime = "go116"
+ available_memory_mb = 128
+ project = var.project_id
+ service_account_email = var.service_account_id
+ source_archive_bucket = var.source_archive_bucket
+ source_archive_object = var.source_archive_object
+ region = var.region
+ ingress_settings = "ALLOW_ALL"
+ # Get the source code of the cloud function as a Zip compression
+ trigger_http = true
+ # Name of the function that will be executed when the Google Cloud Function is triggered
+ entry_point = var.entry_point_names[count.index]
+
+ environment_variables = {
+ DATASTORE_PROJECT_ID=var.project_id
+ GOOGLE_PROJECT_ID=var.project_id
+ PLAYGROUND_ROUTER_HOST=var.pg_router_host
+ }
+
+ timeouts {
+ create = "20m"
+ delete = "20m"
+ }
+
+}
Review Comment:
Looking at this terraform its confusing the purpose of this cloud function. This comment isn't about whether or not to use a cloud function. It's just not very clear. I see that the repositories/client/cloud_functions_client.dart has references that look like HTTP endpoints but apart from that it isn't really clear.
Assuming this is to deploy a REST-like API for http endpoints, it isn't clear what or where the client/server contract is such as an OpenAPI specification. In my experience, things like this work well in the beginning but as the application evolves, inevitable drift happens that breaks and is difficult to troubleshoot later. This is why OpenAPI specifications exist.
##########
learning/tour-of-beam/terraform/cloud_functions/main.tf:
##########
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+resource "google_cloudfunctions_function" "cloud_function" {
+ count = length(var.entry_point_names)
+ name = "${var.environment}_${var.entry_point_names[count.index]}"
+ runtime = "go116"
+ available_memory_mb = 128
+ project = var.project_id
+ service_account_email = var.service_account_id
+ source_archive_bucket = var.source_archive_bucket
+ source_archive_object = var.source_archive_object
+ region = var.region
+ ingress_settings = "ALLOW_ALL"
+ # Get the source code of the cloud function as a Zip compression
+ trigger_http = true
+ # Name of the function that will be executed when the Google Cloud Function is triggered
+ entry_point = var.entry_point_names[count.index]
+
+ environment_variables = {
+ DATASTORE_PROJECT_ID=var.project_id
+ GOOGLE_PROJECT_ID=var.project_id
+ PLAYGROUND_ROUTER_HOST=var.pg_router_host
+ }
+
+ timeouts {
+ create = "20m"
+ delete = "20m"
+ }
+
+}
+
+# Create IAM entry so all users can invoke the function
+resource "google_cloudfunctions_function_iam_member" "invoker" {
+ count = length(google_cloudfunctions_function.cloud_function)
+ project = var.project_id
+ region = var.region
+ cloud_function = google_cloudfunctions_function.cloud_function[count.index].name
+
+ role = "roles/cloudfunctions.invoker"
+ member = "allUsers"
Review Comment:
Is `allUsers` because these cloud functions just serve content?
##########
learning/tour-of-beam/terraform/build.gradle.kts:
##########
@@ -0,0 +1,388 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.pswidersk.gradle.terraform.TerraformTask
+import java.io.ByteArrayOutputStream
+import java.util.regex.Pattern
+
+plugins {
+ id("com.pswidersk.terraform-plugin") version "1.0.0"
+}
+
+terraformPlugin {
+ terraformVersion.set("1.0.9")
+}
+
+/* init Infrastructure for migrate */
+tasks.register<TerraformTask>("terraformInit") {
+ // exec args can be passed by commandline, for example
+ var environment = project.property("project_environment") as String
+ args(
+ "init", "-migrate-state",
+ "-backend-config=./environment/$environment/state.tfbackend",
+ "-var=environment=$environment",
+ if (file("./environment/$environment/terraform.tfvars").exists()) {
+ "-var-file=./environment/$environment/terraform.tfvars"
+ } else {
+ "-no-color"
+ }
+ )
+ }
Review Comment:
Why can't someone just?
```
terraform init -backend-config=./environment/path/to/envirionment/state.tfbackend
```
##########
learning/tour-of-beam/terraform/cloud_functions/variables.tf:
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+variable "service_account_id" {
+ description = "The name of Service Account to run Cloud Function"
Review Comment:
This could probably just be defaulted or put in a common.tfvars file.
##########
learning/tour-of-beam/terraform/cloud_functions/output.tf:
##########
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+output "cloud-function-trigger-url" {
+ value = google_cloudfunctions_function.cloud_function.*.https_trigger_url
Review Comment:
A description might be valuable in documenting what this output is. https://developer.hashicorp.com/terraform/language/values/outputs#description-output-value-documentation
##########
learning/tour-of-beam/terraform/cloud_functions/variables.tf:
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+variable "service_account_id" {
+ description = "The name of Service Account to run Cloud Function"
+}
+
+variable "project_id" {
+ description = "The GCP Project ID of function"
+}
+
+variable "region" {
+ description = "The GCP Region of function"
+}
+
+variable "source_archive_bucket" {
+ description = "The GCS bucket containing the zip archive which contains the function"
+}
+variable "source_archive_object" {
+ description = "The source archive object (file) in archive bucket"
+}
+
+variable "entry_point_names" {
+ type = list
+ default = ["getSdkList", "getContentTree", "getUnitContent", "getUserProgress", "postUnitComplete", "postUserCode", "postDeleteProgress"]
+}
+
+variable "pg_router_host" {
+ description = "Hostname:port of Playground GKE cluster's router grpc workload"
+}
+
+variable "environment" {}
Review Comment:
Could we see a description of what this is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org