You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/12/03 19:23:15 UTC
[GitHub] [beam] emilymye commented on a change in pull request #16039: [Playground][BEAM-12941][Improvement]Add ability to run deployment manually
emilymye commented on a change in pull request #16039:
URL: https://github.com/apache/beam/pull/16039#discussion_r762177355
##########
File path: playground/DEPLOY.md
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+## Manual App Install
+To install Playground applications into existed GCP account you'll need to follow several steps, in this document just simple set of commands will be provided that will guide you over install process.
+
+Before starting that process, you will need to create GCP Artifact Registry instance to store there docker files that would be deployed. Guide to install that registry could be found in [that](https://github.com/apache/beam/blob/master/playground/terraform/README.md) document.
+### Common steps
+First of all you will need to get service account key in json format, that can be done by following [that](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) document. Let's assume that you saved that key into file named key.json.
+Next step is to authentificate your account in GCP docker registry, this can be done by running command:
+```bash
+$ cat /root/dt-key.json | docker login -u _json_key --password-stdin REGISTRY_NAME
Review comment:
Can we recommend following https://cloud.google.com/container-registry/docs/advanced-authentication (more specifically https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud-helper)
##########
File path: playground/DEPLOY.md
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+## Manual App Install
+To install Playground applications into existed GCP account you'll need to follow several steps, in this document just simple set of commands will be provided that will guide you over install process.
+
+Before starting that process, you will need to create GCP Artifact Registry instance to store there docker files that would be deployed. Guide to install that registry could be found in [that](https://github.com/apache/beam/blob/master/playground/terraform/README.md) document.
+### Common steps
+First of all you will need to get service account key in json format, that can be done by following [that](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) document. Let's assume that you saved that key into file named key.json.
+Next step is to authentificate your account in GCP docker registry, this can be done by running command:
+```bash
+$ cat /root/dt-key.json | docker login -u _json_key --password-stdin REGISTRY_NAME
+```
+you will need to replace `REGISTRY_NAME` to actual registry address that can be found in Google Cloud console of via gcloud tool.
+
+Next step is to specify path to json key to be used to deploy applications to Google App Engine. This could be done by command
+```bash
+$ export GOOGLE_APPLICATION_CREDENTIALS=PATH_TO_KEY_JSON_FILE
+```
+replace `PATH_TO_KEY_JSON_FILE` with actual path to key.json file
+
+### Deploy backend application
+
+Now you can start deploying Playground backend application.
+First of all you will need to build and push image to docker registry, this can be done by command
+```bash
+$ ./gradlew playground:backend:containers:java:dockerTagsPush -Pdocker-repository-root='REGISTRY_NAME' -Pbase-image='apache/beam_java8_sdk:BEAM_VERSION' -Pdocker-tag="BACKEND_TAG"
+```
+replace placeholders by
+* `REGISTRY_NAME` with actual registry address, same as was defined in first command(docker login)
+* `BEAM_VERSION` with version of Apache Beam that you want to use
+* `BACKEND_TAG` tag that playground backend docker image will be set and pushed to repo
+
+And final step for deploying backend is deploy via terraform, to run that you will need to run commands
+
+```bash
+$ cd playground/terraform/applications/backend/
+$ terraform init
+$ terraform apply -auto-approve -var="project_id=GCP_PROJECT_ID" -var="docker_registry_address=REGISTRY_NAME" -var="docker_image_tag=BACKEND_TAG" -var="service_name=BACKEND_SERVICE_NAME"
+```
+
+replace placeholdes by
Review comment:
```suggestion
Replace the following variables:
```
##########
File path: playground/DEPLOY.md
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+## Manual App Install
+To install Playground applications into existed GCP account you'll need to follow several steps, in this document just simple set of commands will be provided that will guide you over install process.
+
+Before starting that process, you will need to create GCP Artifact Registry instance to store there docker files that would be deployed. Guide to install that registry could be found in [that](https://github.com/apache/beam/blob/master/playground/terraform/README.md) document.
+### Common steps
+First of all you will need to get service account key in json format, that can be done by following [that](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) document. Let's assume that you saved that key into file named key.json.
+Next step is to authentificate your account in GCP docker registry, this can be done by running command:
+```bash
+$ cat /root/dt-key.json | docker login -u _json_key --password-stdin REGISTRY_NAME
+```
+you will need to replace `REGISTRY_NAME` to actual registry address that can be found in Google Cloud console of via gcloud tool.
+
+Next step is to specify path to json key to be used to deploy applications to Google App Engine. This could be done by command
+```bash
+$ export GOOGLE_APPLICATION_CREDENTIALS=PATH_TO_KEY_JSON_FILE
+```
+replace `PATH_TO_KEY_JSON_FILE` with actual path to key.json file
+
+### Deploy backend application
+
+Now you can start deploying Playground backend application.
+First of all you will need to build and push image to docker registry, this can be done by command
+```bash
+$ ./gradlew playground:backend:containers:java:dockerTagsPush -Pdocker-repository-root='REGISTRY_NAME' -Pbase-image='apache/beam_java8_sdk:BEAM_VERSION' -Pdocker-tag="BACKEND_TAG"
+```
+replace placeholders by
+* `REGISTRY_NAME` with actual registry address, same as was defined in first command(docker login)
Review comment:
```suggestion
* `REGISTRY_NAME`: The AR repository root URL created earlier to host Docker images
```
(I would also maybe change REGISTRY_NAME to REPOSITORY_ROOT or similar
##########
File path: playground/DEPLOY.md
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+## Manual App Install
+To install Playground applications into existed GCP account you'll need to follow several steps, in this document just simple set of commands will be provided that will guide you over install process.
+
+Before starting that process, you will need to create GCP Artifact Registry instance to store there docker files that would be deployed. Guide to install that registry could be found in [that](https://github.com/apache/beam/blob/master/playground/terraform/README.md) document.
+### Common steps
+First of all you will need to get service account key in json format, that can be done by following [that](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) document. Let's assume that you saved that key into file named key.json.
+Next step is to authentificate your account in GCP docker registry, this can be done by running command:
+```bash
+$ cat /root/dt-key.json | docker login -u _json_key --password-stdin REGISTRY_NAME
+```
+you will need to replace `REGISTRY_NAME` to actual registry address that can be found in Google Cloud console of via gcloud tool.
+
+Next step is to specify path to json key to be used to deploy applications to Google App Engine. This could be done by command
+```bash
+$ export GOOGLE_APPLICATION_CREDENTIALS=PATH_TO_KEY_JSON_FILE
+```
+replace `PATH_TO_KEY_JSON_FILE` with actual path to key.json file
+
+### Deploy backend application
+
+Now you can start deploying Playground backend application.
+First of all you will need to build and push image to docker registry, this can be done by command
+```bash
+$ ./gradlew playground:backend:containers:java:dockerTagsPush -Pdocker-repository-root='REGISTRY_NAME' -Pbase-image='apache/beam_java8_sdk:BEAM_VERSION' -Pdocker-tag="BACKEND_TAG"
+```
+replace placeholders by
+* `REGISTRY_NAME` with actual registry address, same as was defined in first command(docker login)
+* `BEAM_VERSION` with version of Apache Beam that you want to use
+* `BACKEND_TAG` tag that playground backend docker image will be set and pushed to repo
+
+And final step for deploying backend is deploy via terraform, to run that you will need to run commands
+
+```bash
+$ cd playground/terraform/applications/backend/
+$ terraform init
+$ terraform apply -auto-approve -var="project_id=GCP_PROJECT_ID" -var="docker_registry_address=REGISTRY_NAME" -var="docker_image_tag=BACKEND_TAG" -var="service_name=BACKEND_SERVICE_NAME"
Review comment:
I think these variables should be one per line
##########
File path: playground/DEPLOY.md
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+## Manual App Install
+To install Playground applications into existed GCP account you'll need to follow several steps, in this document just simple set of commands will be provided that will guide you over install process.
Review comment:
I think maybe we can improve this guide by changing the guide to the following steps (which I think are the requirements for manually installing the Playground app)
1) Building and pushing the backend docker image file to Google Artifact Registry (AR)
2) Building and pushing frontend config files and docker image file to AR
3) Deploy services via terraform.
Where each section describes the commands and variables and links to GCP guides if needed.
##########
File path: playground/DEPLOY.md
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+## Manual App Install
+To install Playground applications into existed GCP account you'll need to follow several steps, in this document just simple set of commands will be provided that will guide you over install process.
+
+Before starting that process, you will need to create GCP Artifact Registry instance to store there docker files that would be deployed. Guide to install that registry could be found in [that](https://github.com/apache/beam/blob/master/playground/terraform/README.md) document.
+### Common steps
+First of all you will need to get service account key in json format, that can be done by following [that](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) document. Let's assume that you saved that key into file named key.json.
+Next step is to authentificate your account in GCP docker registry, this can be done by running command:
+```bash
+$ cat /root/dt-key.json | docker login -u _json_key --password-stdin REGISTRY_NAME
+```
+you will need to replace `REGISTRY_NAME` to actual registry address that can be found in Google Cloud console of via gcloud tool.
Review comment:
Having the doc link (https://cloud.google.com/artifact-registry/docs/manage-repos#view) or example here would be helpful (either the gcloud command like `gcloud artifacts repositories list` or something like `us-central1-docker.pkg.dev/my-project/playground-repository`).
I think it would also be better to have this link to the actual guides from GCP documentation to create AR repositories like https://cloud.google.com/artifact-registry/docs/manage-repos
##########
File path: playground/DEPLOY.md
##########
@@ -0,0 +1,95 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+## Manual App Install
+To install Playground applications into existed GCP account you'll need to follow several steps, in this document just simple set of commands will be provided that will guide you over install process.
+
+Before starting that process, you will need to create GCP Artifact Registry instance to store there docker files that would be deployed. Guide to install that registry could be found in [that](https://github.com/apache/beam/blob/master/playground/terraform/README.md) document.
+### Common steps
+First of all you will need to get service account key in json format, that can be done by following [that](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) document. Let's assume that you saved that key into file named key.json.
+Next step is to authentificate your account in GCP docker registry, this can be done by running command:
+```bash
+$ cat /root/dt-key.json | docker login -u _json_key --password-stdin REGISTRY_NAME
+```
+you will need to replace `REGISTRY_NAME` to actual registry address that can be found in Google Cloud console of via gcloud tool.
+
+Next step is to specify path to json key to be used to deploy applications to Google App Engine. This could be done by command
+```bash
+$ export GOOGLE_APPLICATION_CREDENTIALS=PATH_TO_KEY_JSON_FILE
+```
+replace `PATH_TO_KEY_JSON_FILE` with actual path to key.json file
+
+### Deploy backend application
+
+Now you can start deploying Playground backend application.
+First of all you will need to build and push image to docker registry, this can be done by command
+```bash
+$ ./gradlew playground:backend:containers:java:dockerTagsPush -Pdocker-repository-root='REGISTRY_NAME' -Pbase-image='apache/beam_java8_sdk:BEAM_VERSION' -Pdocker-tag="BACKEND_TAG"
+```
+replace placeholders by
Review comment:
```suggestion
Replace the following variables:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org