You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ruslan-ikhsan (via GitHub)" <gi...@apache.org> on 2023/01/23 13:24:04 UTC
[GitHub] [beam] ruslan-ikhsan opened a new pull request, #25117: Playground ci webhook
ruslan-ikhsan opened a new pull request, #25117:
URL: https://github.com/apache/beam/pull/25117
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1175150714
##########
playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/triggers.tf:
##########
@@ -63,20 +71,26 @@ resource "google_cloudbuild_trigger" "playground_to_gke" {
description = "Creates cloud build manual trigger to deploy Playground to GKE"
source_to_build {
- uri = "https://github.com/beamplayground/deploy-workaround"
+ uri = var.trigger_source_repo
ref = "refs/heads/master"
repo_type = "GITHUB"
}
-
- git_file_source {
- path = ""
- uri = "https://github.com/beamplayground/deploy-workaround"
- revision = "refs/heads/master"
- repo_type = "GITHUB"
+ build {
+ timeout = "7200s"
+ options {
+ logging = "CLOUD_LOGGING_ONLY"
Review Comment:
options:
machineType: E2_HIGHCPU_32
logging: GCS_ONLY
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158462227
##########
playground/infrastructure/cloudbuild/cloudbuild_helm_update.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
Review Comment:
`apt-key` is deprecated. Please follow the official Docker guide for setting up the repo: https://docs.docker.com/engine/install/ubuntu/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158631207
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,244 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Script validates examples against runners (Java, Python, Go) built as local containers from a current branch
+#
+# Command line arguments:
+# LOG_PATH - full path lo a log file. Output is also logged to stdout (Default: /dev/null)
+# BEAM_ROOT_DIR - Path (full) to cloned repo root. Used by ci_cd.py (Default: /workspace/beam)
+# PROJECT_ID - GCP Project ID. Used by ci_cd.py (Default: test)
+# BEAM_CONCURRENCY - Number of examples to run in parallel. Used by ci_cd.py (Default: 4)
+# BEAM_VERSION - version of BEAM SDK to build containers
+# SUBDIRS - array of paths (relative to beam repo root) to search changed examples (Default: ./learning/katas ./examples ./sdks)
+# COMMIT - Git commit hash to build containers from (Default: HEAD)
+# DIFF_BASE - Git branch to compare with $COMMIT to look for changed examples (Default: origin/master)
+# SDKS - Array of SDKS to validate (Default: java python go)
+# ORIGIN - examples origin (Default: PG_EXAMPLES)
+# ALLOWLIST - List of paths (relative to the repo root) not in examples (SUBDIRS) that cause reb
+
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+
+ export "$KEY"="$VALUE"
+done
+
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export BEAM_VERSION=${BEAM_VERSION-"2.44.0"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-"4"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend"}
+
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CILOG keyword to simplify search over the global log
+ echo "CILOG $(date --utc '+%D %T') $1"
+}
+
+
+LogOutput "Input variables:
+ LOG_PATH=$LOG_PATH
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ BEAM_VERSION=$BEAM_VERSION
+ SUBDIRS=$SUBDIRS
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ ORIGIN=$ORIGIN
+ ALLOWLIST=$ALLOWLIST"
+
+# Assigning constant values
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CI
+export SDK_CONFIG="$BEAM_ROOT_DIR/playground/sdks.yaml"
+export BEAM_EXAMPLE_CATEGORIES="$BEAM_ROOT_DIR/playground/categories.yaml"
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+LogOutput "Installing python java8 and dependencies"
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+LogOutput "Installing Python environment"
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install -y --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
Review Comment:
Update pip before using it
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,244 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Script validates examples against runners (Java, Python, Go) built as local containers from a current branch
+#
+# Command line arguments:
+# LOG_PATH - full path lo a log file. Output is also logged to stdout (Default: /dev/null)
+# BEAM_ROOT_DIR - Path (full) to cloned repo root. Used by ci_cd.py (Default: /workspace/beam)
+# PROJECT_ID - GCP Project ID. Used by ci_cd.py (Default: test)
+# BEAM_CONCURRENCY - Number of examples to run in parallel. Used by ci_cd.py (Default: 4)
+# BEAM_VERSION - version of BEAM SDK to build containers
+# SUBDIRS - array of paths (relative to beam repo root) to search changed examples (Default: ./learning/katas ./examples ./sdks)
+# COMMIT - Git commit hash to build containers from (Default: HEAD)
+# DIFF_BASE - Git branch to compare with $COMMIT to look for changed examples (Default: origin/master)
+# SDKS - Array of SDKS to validate (Default: java python go)
+# ORIGIN - examples origin (Default: PG_EXAMPLES)
+# ALLOWLIST - List of paths (relative to the repo root) not in examples (SUBDIRS) that cause reb
+
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+
+ export "$KEY"="$VALUE"
+done
+
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export BEAM_VERSION=${BEAM_VERSION-"2.44.0"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-"4"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend"}
+
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CILOG keyword to simplify search over the global log
+ echo "CILOG $(date --utc '+%D %T') $1"
+}
+
+
+LogOutput "Input variables:
+ LOG_PATH=$LOG_PATH
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ BEAM_VERSION=$BEAM_VERSION
+ SUBDIRS=$SUBDIRS
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ ORIGIN=$ORIGIN
+ ALLOWLIST=$ALLOWLIST"
+
+# Assigning constant values
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CI
+export SDK_CONFIG="$BEAM_ROOT_DIR/playground/sdks.yaml"
+export BEAM_EXAMPLE_CATEGORIES="$BEAM_ROOT_DIR/playground/categories.yaml"
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+LogOutput "Installing python java8 and dependencies"
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+LogOutput "Installing Python environment"
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install -y --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
Review Comment:
Update pip before using it, not after
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158691082
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_infra_part2.yaml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk kubectl > /dev/null
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ git clone --branch merged_pg_gke_deployment
+ https://github.com/akvelon/beam.git
Review Comment:
It should be substitution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1130813519
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
Review Comment:
No need for numbering here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1139905771
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,154 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
Review Comment:
Remove this variable from the script completely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ruslan-ikhsan commented on a diff in pull request #25117: Playground ci webhook
Posted by "ruslan-ikhsan (via GitHub)" <gi...@apache.org>.
ruslan-ikhsan commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138388533
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ 1. PR URL: $_PR_URL;
+ 2. PR Source Repo: $_FORK_REPO;
+ 3. PR Branch: $_PR_BRANCH;
+ 4. PR Commit: $_PR_COMMIT;
+ 5. WebHook Action: $_PR_TYPE;
+ 6. Merged to: $_TARGET_PR_REPO_BRANCH;
+ 7. Playground DNS: $_DNS_NAME; "
+ if [[ "${_PR_BRANCH}" != "playground-ci-webhook" ]]; then
+ exit 0
+ fi
+ if [[ ${_TARGET_PR_REPO_BRANCH} == "apache:master" ]]; then
+ echo "CDLOG $(date --utc '+%D %T').
+
+ Pull Request $_PR_URL has been successfully merged into Apache/Beam Master repository (https://github.com/apache/beam).
+
+ Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git > /dev/null 2>&1
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/akvelon/beam/playground-ci-webhook/playground/infrastructure/cloudbuild/examples_cd.sh
Review Comment:
Replaced with apache/master
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ 1. PR URL: $_PR_URL;
+ 2. PR Source Repo: $_FORK_REPO;
+ 3. PR Branch: $_PR_BRANCH;
+ 4. PR Commit: $_PR_COMMIT;
+ 5. WebHook Action: $_PR_TYPE;
+ 6. Merged to: $_TARGET_PR_REPO_BRANCH;
+ 7. Playground DNS: $_DNS_NAME; "
+ if [[ "${_PR_BRANCH}" != "playground-ci-webhook" ]]; then
Review Comment:
Test entries removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138188356
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ 1. PR URL: $_PR_URL;
+ 2. PR Source Repo: $_FORK_REPO;
+ 3. PR Branch: $_PR_BRANCH;
+ 4. PR Commit: $_PR_COMMIT;
+ 5. WebHook Action: $_PR_TYPE;
+ 6. Merged to: $_TARGET_PR_REPO_BRANCH;
+ 7. Playground DNS: $_DNS_NAME; "
+ if [[ "${_PR_BRANCH}" != "playground-ci-webhook" ]]; then
Review Comment:
Remove test entries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138187374
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
Review Comment:
This a constant. No need to read it from command line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1178852651
##########
playground/infrastructure/cloudbuild/playground_cd_examples.sh:
##########
@@ -0,0 +1,187 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export FORCE_CD=${FORCE_CD-'false'}
+
+# Playground FQDN
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+# GCP Datastore namespace to load examples to
+if [[ -z "${DATASTORE_NAMESPACE}" ]]; then
+ echo "DATASTORE_NAMESPACE parameter was not set. Exiting"
+ exit 1
+fi
+# Master branch commit after merge
+if [[ -z "${MERGE_COMMIT}" ]]; then
+ echo "MERGE_COMMIT paramter was not set. Exiting"
+ exit 1
+fi
+if [[ "${FORCE_CD}" != "false" && "${FORCE_CD}" != "true" ]]; then
+ echo "FORCE_CD paramter must be either 'true' or 'false'. Exiting"
+ exit 1
+fi
+
+# Constants
+export STEP=CD
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ MERGE_COMMIT=$MERGE_COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ DATASTORE_NAMESPACE=$DATASTORE_NAMESPACE
+ FORCE_CD=$FORCE_CD"
+
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+LogOutput "Installing python and dependencies."
+# set -e # Exit immediately if any command fails
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install -y --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null 2>&1
Review Comment:
`apt-get install python-is-python3`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158458240
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_infra_part1.yaml:
##########
@@ -0,0 +1,93 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
Review Comment:
`apt-key` is deprecated. Please add gpg file to keyring directly like
```sh
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo tee /usr/share/keyrings/cloud.google.gpg
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158603655
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"learning/katas sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CD
+
+LogOutput "Installing python and dependencies."
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install -y --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null 2>&1
+apt install -y python3.8-venv > /dev/null 2>&1
+pip install -r /workspace/beam/playground/infrastructure/requirements.txt > /dev/null 2>&1
+
+LogOutput "Python and dependencies have been successfully installed."
+
+LogOutput "Checking what files were changed in the PR."
+
+git fetch --all > /dev/null 2>&1
+
+diff_log=$(git diff --name-only $DIFF_BASE...forked/$SOURCE_BRANCH)
+diff=($(echo "$diff_log" | tr '\n' ' '))
+LogOutput "List of changed files in PR $SOURCE_BRANCH merging into $DIFF_BASE are:
+$diff_log"
+
+LogOutput "Looking for changes that require CD validation for [$SDKS] SDKs"
+allowlist_array=($ALLOWLIST)
+for sdk in $SDKS
+do
+ eval "example_for_${sdk}_changed"='False'
+ LogOutput "------------------Starting checker.py for SDK_${sdk^^}------------------"
+ cd $BEAM_ROOT_DIR/playground/infrastructure
+ python3 checker.py \
+ --verbose \
+ --sdk SDK_"${sdk^^}" \
+ --allowlist "" \
+ --paths "${diff[@]}"
+ checker_status=$?
+ if [ $checker_status -eq 0 ]; then
+ LogOutput "Checker found changed examples for SDK_${sdk^^}"
+ eval "example_for_${sdk}_changed"='True'
+ elif [ $checker_status -eq 11 ]; then
+ LogOutput "Checker did not find any changed examples for SDK_${sdk^^}"
+ eval "example_for_${sdk}_changed"='False'
+ continue
+ else
+ LogOutput "Error: Checker is broken. Exiting the script."
+ exit 1
+ fi
+
+ result=$(eval echo '$'"example_for_${sdk}_changed")
+
+ if [[ $result == True ]]; then
+ LogOutput "Running ci_cd.py for SDK $sdk"
+
+ export SERVER_ADDRESS=https://${sdk}.${DNS_NAME}
+ python3 ci_cd.py \
+ --datastore-project ${PROJECT_ID} \
+ --namespace ${NAMESPACE} \
+ --step ${STEP} \
+ --sdk SDK_"${sdk^^}" \
+ --origin ${ORIGIN} \
+ --subdirs ${SUBDIRS}
+ if [ $? -eq 0 ]; then
+ LogOutput "Examples for $sdk SDK have been successfully deployed."
+ else
+ LogOutput "Examples for $sdk SDK were not deployed. Please see the logs."
+ fi
+ fi
+done
Review Comment:
Add end of line at the end of the file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158594499
##########
playground/infrastructure/cloudbuild/cloudbuild_helm_update.yaml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
Review Comment:
What's the rationale behind redirecting all output to `/dev/null`? Here and everywhere else
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158684024
##########
playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/variables.tf:
##########
@@ -16,76 +16,72 @@
# under the License.
variable "project_id" {
- type = string
description = "The ID of the Google Cloud project within which resources are provisioned"
}
-variable "region" {
- type = string
- description = "The Google Cloud Platform (GCP) region (For example: us-central1) where Cloud Build triggers will be created at"
-}
-
-variable "infra_trigger_name" {
- type = string
+variable "pg_infra_trigger_name" {
description = "The name of the trigger that will deploy Playground infrastructure"
- default = "playground-infrastructure-trigger"
+ default = "Playground-infrastructure-trigger"
Review Comment:
Rename triggers Display name. E.g.:
Initialize Playground environment
Deploy Playground environment
Update Playground environment
Validate examples (CI)
Deploy examples (CD)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ruslan-ikhsan commented on a diff in pull request #25117: Playground ci webhook
Posted by "ruslan-ikhsan (via GitHub)" <gi...@apache.org>.
ruslan-ikhsan commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1155911449
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,154 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1179268440
##########
playground/terraform/infrastructure/cloudbuild-manual-setup/README.md:
##########
@@ -31,18 +31,21 @@ Cloud Build triggers created by terraform scripts from this directory automate s
- Cloud Build Editor
- Security Admin
- Service Account User
+ - Secret Manager Admin
- [gcloud CLI](https://cloud.google.com/sdk/docs/install-sdk)
-- An existing GCP Bucket to save Terraform state - `state-bucket`
+- An existing GCS Bucket to save Terraform state - `state-bucket`
+- An existing GCS Bucket to store CI privae logs - `private-logs-bucket`
Review Comment:
private
Also fo public logs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1139904613
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL;
+ PR Source Repo: $_FORK_REPO;
+ PR Branch: $_PR_BRANCH;
+ PR Commit: $_PR_COMMIT;
+ WebHook Action: $_PR_TYPE;
+ Merged to: $_TARGET_PR_REPO_BRANCH;
+ Playground DNS: $_DNS_NAME; "
+ if [[ ${_TARGET_PR_REPO_BRANCH} != "apache:master" ]]; then
+ echo "CDLOG Merge to branch apache/${_TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == "closed" ]] && [[ ${_MERGE_STATUS} == "true" ]]; then
+
+ echo "CDLOG Pull Request $_PR_URL has been successfully closed and merged into Apache/Beam GitHub repository. Continuing the process."
+
+ echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git > /dev/null 2>&1
Review Comment:
Do you need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1178845346
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_ci_examples.sh:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#!/usr/bin/env bash
+echo "Search for CILOG keyword to find valuable logs entries"
+echo "CILOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL:$_PR_URL
+ PR Source Repo: $_FORK_REPO
+ PR Branch: $_PR_BRANCH
+ PRCommit: $_PR_COMMIT
+ WebHook Action: $_PR_TYPE
+ Merge to: $_BASE_REF"
+
+
+if [[ "${_BASE_REF}" != "master" ]]; then
+ echo "CILOG Merge to branch apache/${_BASE_REF}. Exiting"
+ exit 0
+fi
+
+if [[ ${_PR_TYPE} == @(opened|synchronize) ]]; then
+
+ echo "CILOG See also public logs: ${_PUBLIC_LOG_URL}"
+
+ echo "CILOG $(date --utc '+%D %T') Examples validation (CI) has started"
+
+ echo "$(date --utc '+%D %T') Examples validation (CI) has started" >> ${_PUBLIC_LOG_LOCAL}
+
+ apt update > /dev/null
+
+ apt-get install -y git curl apt-transport-https ca-certificates gnupg > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
Review Comment:
`apt-key` is deprecated. Please do not use `apt-key`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1178840291
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_cd_examples.sh:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#!/usr/bin/env bash
+echo "Search for CDLOG keyword to find valuable logs entries"
+echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $PR_URL
+ Merged to: $TARGET_PR_REPO_BRANCH
+ WebHook Action: $PR_TYPE
+ Merge Status: $MERGE_STATUS
+ Merge commit: $MERGE_COMMIT
+ Playground DNS: $DNS_NAME
+ Datastore namespace: $DATASTORE_NAMESPACE
+ Exampes Origin: $ORIGIN
+ Examples Subdirs: $SUBDIRS
+ SDKs: $SDKS
+ CD Script Concurrency: $BEAM_CONCURRENCY"
+
+if [ -z "$DNS_NAME" ] || [ -z "$DATASTORE_NAMESPACE" ]; then
+ echo "DNS_NAME and DATASTORE_NAMESPACE substitutions must be set in the triger"
+ exit 1
+fi
+
+
+if [ "${TARGET_PR_REPO_BRANCH}" != "apache:master" ]; then
+ echo "CDLOG Merging not into the master, but into the branch apache/${TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+fi
+
+
+if [ ${PR_TYPE} != "closed" ] || [ ${MERGE_STATUS} != "true" ];
+then
+ echo "CDLOG $(date --utc '+%D %T') PR in $PR_URL for the COMMIT $PR_COMMIT is not in closed state or not merged into the Apache/Beam master repo."
+ exit 0
+fi
+
+
+echo "CDLOG Pull Request $PR_URL has been successfully merged into Apache Beam GitHub repository. Continuing the process."
+
+echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+
+apt update > /dev/null 2>&1
+
+
+apt install -y git curl > /dev/null 2>&1
+
+
+apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+
+echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+
+/etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+
+apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+echo "CDLOG $(date --utc '+%D %T') Starting deployment script"
+
+git clone --branch master https://github.com/apache/beam.git
+
+chmod +x ${CD_SCRIPT_PATH}
+
+bash -c "${CD_SCRIPT_PATH}"
Review Comment:
What's the rationale behind using `-c`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158458240
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_infra_part1.yaml:
##########
@@ -0,0 +1,93 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
Review Comment:
`apt-key` is deprecated. Please add gpg keyring file directly like
```sh
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo tee /usr/share/keyrings/cloud.google.gpg
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158632534
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,244 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Script validates examples against runners (Java, Python, Go) built as local containers from a current branch
+#
+# Command line arguments:
+# LOG_PATH - full path lo a log file. Output is also logged to stdout (Default: /dev/null)
+# BEAM_ROOT_DIR - Path (full) to cloned repo root. Used by ci_cd.py (Default: /workspace/beam)
+# PROJECT_ID - GCP Project ID. Used by ci_cd.py (Default: test)
+# BEAM_CONCURRENCY - Number of examples to run in parallel. Used by ci_cd.py (Default: 4)
+# BEAM_VERSION - version of BEAM SDK to build containers
+# SUBDIRS - array of paths (relative to beam repo root) to search changed examples (Default: ./learning/katas ./examples ./sdks)
+# COMMIT - Git commit hash to build containers from (Default: HEAD)
+# DIFF_BASE - Git branch to compare with $COMMIT to look for changed examples (Default: origin/master)
+# SDKS - Array of SDKS to validate (Default: java python go)
+# ORIGIN - examples origin (Default: PG_EXAMPLES)
+# ALLOWLIST - List of paths (relative to the repo root) not in examples (SUBDIRS) that cause reb
+
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+
+ export "$KEY"="$VALUE"
+done
+
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export BEAM_VERSION=${BEAM_VERSION-"2.44.0"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-"4"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend"}
+
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CILOG keyword to simplify search over the global log
+ echo "CILOG $(date --utc '+%D %T') $1"
+}
+
+
+LogOutput "Input variables:
+ LOG_PATH=$LOG_PATH
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ BEAM_VERSION=$BEAM_VERSION
+ SUBDIRS=$SUBDIRS
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ ORIGIN=$ORIGIN
+ ALLOWLIST=$ALLOWLIST"
+
+# Assigning constant values
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CI
+export SDK_CONFIG="$BEAM_ROOT_DIR/playground/sdks.yaml"
+export BEAM_EXAMPLE_CATEGORIES="$BEAM_ROOT_DIR/playground/categories.yaml"
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+LogOutput "Installing python java8 and dependencies"
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+LogOutput "Installing Python environment"
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install -y --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null
+apt install -y python3.8-venv > /dev/null
+pip install -r playground/infrastructure/requirements.txt > /dev/null
Review Comment:
This whole section seems to be exactly the same as in the other script - can we move it some common place?
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,244 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Script validates examples against runners (Java, Python, Go) built as local containers from a current branch
+#
+# Command line arguments:
+# LOG_PATH - full path lo a log file. Output is also logged to stdout (Default: /dev/null)
+# BEAM_ROOT_DIR - Path (full) to cloned repo root. Used by ci_cd.py (Default: /workspace/beam)
+# PROJECT_ID - GCP Project ID. Used by ci_cd.py (Default: test)
+# BEAM_CONCURRENCY - Number of examples to run in parallel. Used by ci_cd.py (Default: 4)
+# BEAM_VERSION - version of BEAM SDK to build containers
+# SUBDIRS - array of paths (relative to beam repo root) to search changed examples (Default: ./learning/katas ./examples ./sdks)
+# COMMIT - Git commit hash to build containers from (Default: HEAD)
+# DIFF_BASE - Git branch to compare with $COMMIT to look for changed examples (Default: origin/master)
+# SDKS - Array of SDKS to validate (Default: java python go)
+# ORIGIN - examples origin (Default: PG_EXAMPLES)
+# ALLOWLIST - List of paths (relative to the repo root) not in examples (SUBDIRS) that cause reb
+
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+
+ export "$KEY"="$VALUE"
+done
+
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export BEAM_VERSION=${BEAM_VERSION-"2.44.0"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-"4"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend"}
+
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CILOG keyword to simplify search over the global log
+ echo "CILOG $(date --utc '+%D %T') $1"
+}
+
+
+LogOutput "Input variables:
+ LOG_PATH=$LOG_PATH
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ BEAM_VERSION=$BEAM_VERSION
+ SUBDIRS=$SUBDIRS
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ ORIGIN=$ORIGIN
+ ALLOWLIST=$ALLOWLIST"
+
+# Assigning constant values
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CI
+export SDK_CONFIG="$BEAM_ROOT_DIR/playground/sdks.yaml"
+export BEAM_EXAMPLE_CATEGORIES="$BEAM_ROOT_DIR/playground/categories.yaml"
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+LogOutput "Installing python java8 and dependencies"
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+LogOutput "Installing Python environment"
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install -y --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null
+apt install -y python3.8-venv > /dev/null
+pip install -r playground/infrastructure/requirements.txt > /dev/null
Review Comment:
This whole section seems to be exactly the same as in the other script - can we move it to some common place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158628828
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"learning/katas sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CD
+
+LogOutput "Installing python and dependencies."
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install -y --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null 2>&1
+apt install -y python3.8-venv > /dev/null 2>&1
+pip install -r /workspace/beam/playground/infrastructure/requirements.txt > /dev/null 2>&1
+
+LogOutput "Python and dependencies have been successfully installed."
+
+LogOutput "Checking what files were changed in the PR."
+
+git fetch --all > /dev/null 2>&1
+
+diff_log=$(git diff --name-only $DIFF_BASE...forked/$SOURCE_BRANCH)
+diff=($(echo "$diff_log" | tr '\n' ' '))
+LogOutput "List of changed files in PR $SOURCE_BRANCH merging into $DIFF_BASE are:
+$diff_log"
+
+LogOutput "Looking for changes that require CD validation for [$SDKS] SDKs"
+allowlist_array=($ALLOWLIST)
+for sdk in $SDKS
+do
+ eval "example_for_${sdk}_changed"='False'
+ LogOutput "------------------Starting checker.py for SDK_${sdk^^}------------------"
+ cd $BEAM_ROOT_DIR/playground/infrastructure
+ python3 checker.py \
+ --verbose \
+ --sdk SDK_"${sdk^^}" \
+ --allowlist "" \
+ --paths "${diff[@]}"
+ checker_status=$?
+ if [ $checker_status -eq 0 ]; then
+ LogOutput "Checker found changed examples for SDK_${sdk^^}"
+ eval "example_for_${sdk}_changed"='True'
+ elif [ $checker_status -eq 11 ]; then
+ LogOutput "Checker did not find any changed examples for SDK_${sdk^^}"
+ eval "example_for_${sdk}_changed"='False'
+ continue
+ else
+ LogOutput "Error: Checker is broken. Exiting the script."
+ exit 1
+ fi
+
+ result=$(eval echo '$'"example_for_${sdk}_changed")
+
+ if [[ $result == True ]]; then
+ LogOutput "Running ci_cd.py for SDK $sdk"
+
+ export SERVER_ADDRESS=https://${sdk}.${DNS_NAME}
+ python3 ci_cd.py \
Review Comment:
If you are installing custom version of python (3.8) please invoke the desired version directly. As written here, it's not clear what version of python executable will actually be executed. If everything works fine with system-default python, I don't see good reasons for installing 3.8, as it only increases build time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158547976
##########
playground/infrastructure/cloudbuild/cloudbuild_helm_update.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk > /dev/null
+
+ git clone --branch merged_pg_gke_deployment
+ https://github.com/akvelon/beam.git
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ cd beam
+
+ mkdir playground/terraform/environment/myenv
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/terraform.tfvars
+ playground/terraform/environment/myenv/terraform.tfvars
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/state.tfbackend
+ playground/terraform/environment/myenv/state.tfbackend
+
+ cat playground/terraform/environment/myenv/terraform.tfvars
+
+ cat playground/terraform/environment/myenv/state.tfbackend
+
+ gcloud auth configure-docker us-east1-docker.pkg.dev
+
+ gcloud container clusters get-credentials --region us-east1-b
+ playground-gke --project cloudbuildtest-382210
+
+ ./gradlew playground:terraform:gkebackend -Pproject_environment=myenv
+ -Pdocker-tag=$_TAG_NAME -Pdns-name=$_DNS_NAME -Psdk-tag=$_SDK_TAG
Review Comment:
This is not Bash script. The yaml syntax is corrected by UI in GCP console. It automatically breaks command to lines. Command separated by empty lines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158691464
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_infra_part2.yaml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk kubectl > /dev/null
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ git clone --branch merged_pg_gke_deployment
Review Comment:
master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158523000
##########
playground/infrastructure/cloudbuild/cloudbuild_helm_update.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk > /dev/null
+
+ git clone --branch merged_pg_gke_deployment
+ https://github.com/akvelon/beam.git
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ cd beam
+
+ mkdir playground/terraform/environment/myenv
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/terraform.tfvars
+ playground/terraform/environment/myenv/terraform.tfvars
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/state.tfbackend
+ playground/terraform/environment/myenv/state.tfbackend
+
+ cat playground/terraform/environment/myenv/terraform.tfvars
+
+ cat playground/terraform/environment/myenv/state.tfbackend
+
+ gcloud auth configure-docker us-east1-docker.pkg.dev
+
+ gcloud container clusters get-credentials --region us-east1-b
+ playground-gke --project cloudbuildtest-382210
+
+ ./gradlew playground:terraform:gkebackend -Pproject_environment=myenv
+ -Pdocker-tag=$_TAG_NAME -Pdns-name=$_DNS_NAME -Psdk-tag=$_SDK_TAG
Review Comment:
i don't quite understand how these multiline commands are supposed to work given that there is no escaping `\` of the line break?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158626955
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"learning/katas sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CD
+
+LogOutput "Installing python and dependencies."
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
Review Comment:
Is system-provided python not suitable for us?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158613919
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,244 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Script validates examples against runners (Java, Python, Go) built as local containers from a current branch
+#
+# Command line arguments:
+# LOG_PATH - full path lo a log file. Output is also logged to stdout (Default: /dev/null)
+# BEAM_ROOT_DIR - Path (full) to cloned repo root. Used by ci_cd.py (Default: /workspace/beam)
+# PROJECT_ID - GCP Project ID. Used by ci_cd.py (Default: test)
+# BEAM_CONCURRENCY - Number of examples to run in parallel. Used by ci_cd.py (Default: 4)
+# BEAM_VERSION - version of BEAM SDK to build containers
+# SUBDIRS - array of paths (relative to beam repo root) to search changed examples (Default: ./learning/katas ./examples ./sdks)
+# COMMIT - Git commit hash to build containers from (Default: HEAD)
+# DIFF_BASE - Git branch to compare with $COMMIT to look for changed examples (Default: origin/master)
+# SDKS - Array of SDKS to validate (Default: java python go)
+# ORIGIN - examples origin (Default: PG_EXAMPLES)
+# ALLOWLIST - List of paths (relative to the repo root) not in examples (SUBDIRS) that cause reb
+
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+
+ export "$KEY"="$VALUE"
+done
+
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export BEAM_VERSION=${BEAM_VERSION-"2.44.0"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-"4"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend"}
+
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CILOG keyword to simplify search over the global log
+ echo "CILOG $(date --utc '+%D %T') $1"
+}
+
+
+LogOutput "Input variables:
+ LOG_PATH=$LOG_PATH
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ BEAM_VERSION=$BEAM_VERSION
+ SUBDIRS=$SUBDIRS
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ ORIGIN=$ORIGIN
+ ALLOWLIST=$ALLOWLIST"
+
+# Assigning constant values
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CI
+export SDK_CONFIG="$BEAM_ROOT_DIR/playground/sdks.yaml"
+export BEAM_EXAMPLE_CATEGORIES="$BEAM_ROOT_DIR/playground/categories.yaml"
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+LogOutput "Installing python java8 and dependencies"
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+LogOutput "Installing Python environment"
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install -y --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null
+apt install -y python3.8-venv > /dev/null
+pip install -r playground/infrastructure/requirements.txt > /dev/null
+
+LogOutput "Installing JDK and Gradle"
+apt-get install openjdk-8-jdk -y > /dev/null
+curl -L https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip -o gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+unzip gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+export PATH=$PATH:gradle-${GRADLE_VERSION}/bin > /dev/null
+
+LogOutput "Installing GO"
+curl -OL https://golang.org/dl/go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+tar -C /usr/local -xvf go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+export PATH=$PATH:/usr/local/go/bin > /dev/null
+# required to build beam/sdks/go container
+export GOMODCACHE="$HOME/go/pkg/mod"
+
+LogOutput "Installing Docker"
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - > /dev/null
+add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+apt update > /dev/null && apt install -y docker-ce > /dev/null
+
+tag_name=$(git tag --points-at $COMMIT)
+if [ "$tag_name" ]
+then
+ LogOutput "Commit $COMMIT is tagged with $tag_name"
+ DOCKERTAG=$tag_name
+else
+ LogOutput "Commit $COMMIT is not tagged"
+ DOCKERTAG=$COMMIT
+fi
+LogOutput "Docker tag for containers: $DOCKERTAG"
+
+LogOutput "git fetch -all"
+git fetch --all > /dev/null
+LogOutput "git checkout $COMMIT"
+git checkout $COMMIT
+if [ $? -ne 0 ]; then
+ LogOutput "Can't checkout to $COMMIT. Exiting script"
+ exit 1
+fi
+
+diff_log=$(git diff --name-only $DIFF_BASE...$COMMIT)
+diff=($(echo "$diff_log" | tr '\n' ' '))
+LogOutput "Discovered changes introduced by $COMMIT relative to $DIFF_BASE in files:
+$diff_log"
+
+LogOutput "Looking for changes that require CI validation for [$SDKS] SDKs"
+allowlist_array=($ALLOWLIST)
+for sdk in $SDKS
+do
+ eval "ci_${sdk}_passed"='False'
+ example_has_changed="UNKNOWN"
+ LogOutput "------------------Starting checker.py for SDK_${sdk^^}------------------"
+ cd $BEAM_ROOT_DIR/playground/infrastructure
+ python3 checker.py \
+ --verbose \
+ --sdk SDK_"${sdk^^}" \
+ --allowlist "${allowlist_array[@]}" \
+ --paths "${diff[@]}" >> ${LOG_PATH} 2>&1
+ checker_status=$?
+ cd $BEAM_ROOT_DIR
+ if [ $checker_status -eq 0 ]
+ then
+ LogOutput "Checker found changed examples for SDK_${sdk^^}"
+ example_has_changed=True
+ elif [ $checker_status -eq 11 ]
+ then
+ LogOutput "Checker did not find any changed examples for SDK_${sdk^^}"
+ example_has_changed=False
+ else
+ LogOutput "Error: Checker is broken. Exiting the script."
+ exit 1
+ fi
+
+ #Nothing to check
+ if [[ $example_has_changed != "True" ]]
+ then
+ LogOutput "No changes require validation for SDK_${sdk^^}"
+ continue
+ fi
+
+ # docker_options="-Psdk-tag=${DOCKERTAG}"
+ sdk_tag=$BEAM_VERSION
+
+ # Special cases for Python and Java
+ if [ "$sdk" == "python" ]
+ then
+ # Build fails without docker-pull-licenses=true in Cloud Build
+ LogOutput "Building Python base image container apache/beam_python3.7_sdk:$DOCKERTAG -Pdocker-pull-licenses=true"
Review Comment:
We updated base image for python containers to `3.10` here: https://github.com/apache/beam/pull/25992. Please update this script accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158618689
##########
playground/infrastructure/cloudbuild/playground_ci_stable.yaml:
##########
@@ -0,0 +1,112 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ echo "Search for CILOG keyword to find valuable logs entries"
+
+ echo "CILOG $(date --utc '+%D %T') Trigger run inputs: PR URL:
+ $_PR_URL; PR Source Repo: $_FORK_REPO; PR Branch: $_PR_BRANCH; PR
+ Commit: $_PR_COMMIT; WebHook Action: $_PR_TYPE; Merge to: $_BASE_REF;"
+
+ if [[ "${_BASE_REF}" != "master" ]]; then
+ echo "CILOG Merging not into master, but into the branch apache/${_BASE_REF}. Exiting."
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == @(opened|synchronize) ]]; then
+
+ echo "CILOG See also public logs: ${_PUBLIC_LOG_URL}"
+
+ echo "CILOG $(date --utc '+%D %T') Examples validation (CI) has started"
+
+ echo "$(date --utc '+%D %T') Examples validation (CI) has started" >> ${_PUBLIC_LOG_LOCAL}
+
+ apt update > /dev/null
+
+ apt-get install -y git curl apt-transport-https ca-certificates gnupg > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null
+
+ git clone --branch master https://github.com/apache/beam.git
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/apache/beam/master/playground/infrastructure/cloudbuild/examples_ci.sh
Review Comment:
Please fetch this file from the commit from which the build is started, otherwise it can lead to very confusing behaviour or unanticipated breakages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158600314
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"learning/katas sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CD
+
+LogOutput "Installing python and dependencies."
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
Review Comment:
Again, what is the rationale behind silencing the output of all commands? This will make debugging very hard
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158691082
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_infra_part2.yaml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk kubectl > /dev/null
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ git clone --branch merged_pg_gke_deployment
+ https://github.com/akvelon/beam.git
Review Comment:
It's fine. But it should be apache, not akvelon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138193338
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ 1. PR URL: $_PR_URL;
+ 2. PR Source Repo: $_FORK_REPO;
+ 3. PR Branch: $_PR_BRANCH;
+ 4. PR Commit: $_PR_COMMIT;
+ 5. WebHook Action: $_PR_TYPE;
+ 6. Merged to: $_TARGET_PR_REPO_BRANCH;
+ 7. Playground DNS: $_DNS_NAME; "
+ if [[ "${_PR_BRANCH}" != "playground-ci-webhook" ]]; then
+ exit 0
+ fi
+ if [[ ${_TARGET_PR_REPO_BRANCH} == "apache:master" ]]; then
Review Comment:
Reverse the condition. Just exit the script if it's not master bracnh. No need to create if/else trees
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1139903494
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL;
+ PR Source Repo: $_FORK_REPO;
Review Comment:
Why do you need fork if the change merged to master?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1139925305
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,154 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+if [[ -z "${PROJECT_ID}" ]]; then
+ echo "PROJECT_ID is empty or not set. Exiting"
+ exit 1
+fi
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=$STEP
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+LogOutput "Installing python and dependencies"
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install -y --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null 2>&1
+apt install -y python3.8-venv > /dev/null 2>&1
+pip install -r /workspace/beam/playground/infrastructure/requirements.txt > /dev/null 2>&1
+
+LogOutput "All packages and dependencies have been successfully installed. Starting Playground examples Deployment to https://${DNS_NAME}."
+
+LogOutput "Checking changed files in the PR"
+
+diff=($(git diff --name-only $DIFF_BASE...$SOURCE_BRANCH | tr '\n' ' '))
+
+for file in "${diff[@]}"; do
+ if echo $file == *learning/katas/ || echo $file == *examples/* || echo $file == *sdks/*
Review Comment:
You also need to check if the change is related to an example with tags. Reload only examples for a specific affected SDK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1139947283
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,154 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+if [[ -z "${PROJECT_ID}" ]]; then
+ echo "PROJECT_ID is empty or not set. Exiting"
+ exit 1
+fi
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=$STEP
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+LogOutput "Installing python and dependencies"
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install -y --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null 2>&1
+apt install -y python3.8-venv > /dev/null 2>&1
+pip install -r /workspace/beam/playground/infrastructure/requirements.txt > /dev/null 2>&1
+
+LogOutput "All packages and dependencies have been successfully installed. Starting Playground examples Deployment to https://${DNS_NAME}."
+
+LogOutput "Checking changed files in the PR"
+
+diff=($(git diff --name-only $DIFF_BASE...$SOURCE_BRANCH | tr '\n' ' '))
+
+for file in "${diff[@]}"; do
Review Comment:
You are reloading all SDKs for each changed file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1178852330
##########
playground/infrastructure/cloudbuild/playground_cd_examples.sh:
##########
@@ -0,0 +1,187 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export FORCE_CD=${FORCE_CD-'false'}
+
+# Playground FQDN
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+# GCP Datastore namespace to load examples to
+if [[ -z "${DATASTORE_NAMESPACE}" ]]; then
+ echo "DATASTORE_NAMESPACE parameter was not set. Exiting"
+ exit 1
+fi
+# Master branch commit after merge
+if [[ -z "${MERGE_COMMIT}" ]]; then
+ echo "MERGE_COMMIT paramter was not set. Exiting"
+ exit 1
+fi
+if [[ "${FORCE_CD}" != "false" && "${FORCE_CD}" != "true" ]]; then
+ echo "FORCE_CD paramter must be either 'true' or 'false'. Exiting"
+ exit 1
+fi
+
+# Constants
+export STEP=CD
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ MERGE_COMMIT=$MERGE_COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ DATASTORE_NAMESPACE=$DATASTORE_NAMESPACE
+ FORCE_CD=$FORCE_CD"
+
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+LogOutput "Installing python and dependencies."
+# set -e # Exit immediately if any command fails
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
Review Comment:
There is no reason to use `python3.8`. System python will be sufficient
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1104656031
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,165 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Added set -x to show output into cloud build console to the bucket
+set -x
+
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+#Install python java8 and dependencies
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+# Env configuration commands
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null
+apt install python3.8-venv > /dev/null
+pip install -r playground/infrastructure/requirements.txt > /dev/null
+
+# Install jdk and gradle
+apt-get install openjdk-8-jdk -y > /dev/null
+curl -L https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip -o gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+unzip gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+export PATH=$PATH:gradle-${GRADLE_VERSION}/bin > /dev/null
+
+# Install go
+curl -OL https://golang.org/dl/go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+tar -C /usr/local -xvf go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+export PATH=$PATH:/usr/local/go/bin > /dev/null
+
+# Install Docker
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - > /dev/null
+add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+apt update > /dev/null && apt install -y docker-ce > /dev/null
+
+# Assigning required values for CI_CD.py script
+export \
+ORIGIN=PG_EXAMPLES \
+STEP=CI \
+SUBDIRS="./learning/katas ./examples ./sdks" \
+GOOGLE_CLOUD_PROJECT=${PROJECT_ID} \
+BEAM_ROOT_DIR="../.." \
+SDK_CONFIG="../sdks.yaml" \
+BEAM_EXAMPLE_CATEGORIES="../categories.yaml" \
+BEAM_CONCURRENCY=4 \
+BEAM_VERSION=2.43.0 \
+sdks=("java" "python" "go") \
+allowlist=("playground/infrastructure" "playground/backend")
+
+# Check whether commit is tagged
+tag_name=$(git tag --points-at $commit_sha)
+
+# Get diff
+if [ -z $base_ref ] || [ $base_ref == "master" ]
+then
+ base_ref=origin/master
+fi
+diff=$(git diff --name-only $base_ref $commit_sha | tr '\n' ' ')
+
+# Check if there are Examples as
+for sdk in "${sdks[@]}"
+do
+ cd playground/infrastructure
+ python3 checker.py \
+ --verbose \
+ --sdk SDK_"${sdk^^}" \
+ --allowlist "${allowlist[@]}" \
+ --paths ${diff}
+ checker_status=$?
+ cd -
+ if [ $checker_status -eq 0 ]
+ then
+ echo "Checker has found changed examples for ${sdk^^}" >> ${1}
+ example_has_changed=True
+ elif [ $checker_status -eq 11 ]
+ then
+ echo "Checker has not found changed examples for ${sdk^^}" >> ${1}
+ example_has_changed=False
+ else
+ echo "Error: Checker is broken" >> ${1}
+ exit 1
+ fi
+
+# Run main logic if examples have been changed
+ if [[ $example_has_changed == "True" ]]
+ then
+ if [ -z "${tag_name}" ] && [ "${commit_sha}" ]
+ then
+ DOCKERTAG=${commit_sha}
+ elif [ "${tag_name}" ] && [ "${commit_sha}" ]
+ then
+ DOCKERTAG=${tag_name}
+ elif [ "${tag_name}" ] && [ -z "${commit_sha}" ]
+ then
+ DOCKERTAG=${tag_name}
+ elif [ -z "${tag_name}" ] && [ -z "${commit_sha}" ]
+ then
+ echo "Error: DOCKERTAG is empty"
+ exit 1
+ fi
+
+ if [ "$sdk" == "python" ]
+ then
+ # builds apache/beam_python3.7_sdk:$DOCKERTAG image
+ ./gradlew -i :sdks:python:container:py37:docker -Pdocker-tag=${DOCKERTAG}
+ # and set SDK_TAG to DOCKERTAG so that the next step would find it
+ SDK_TAG=${DOCKERTAG}
+ else
+ unset SDK_TAG
+ fi
+
+ opts=" -Pdocker-tag=${DOCKERTAG}"
+ if [ -n "$SDK_TAG" ]
+ then
+ opts="${opts} -Psdk-tag=${SDK_TAG}"
+ fi
+
+ if [ "$sdk" == "java" ]
+ then
+ # Java uses a fixed BEAM_VERSION
+ opts="$opts -Pbase-image=apache/beam_java8_sdk:2.43.0"
Review Comment:
Replace to an optional parameter for a script with default value "2.43.0".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1178836936
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_cd_examples.sh:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#!/usr/bin/env bash
+echo "Search for CDLOG keyword to find valuable logs entries"
+echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $PR_URL
+ Merged to: $TARGET_PR_REPO_BRANCH
+ WebHook Action: $PR_TYPE
+ Merge Status: $MERGE_STATUS
+ Merge commit: $MERGE_COMMIT
+ Playground DNS: $DNS_NAME
+ Datastore namespace: $DATASTORE_NAMESPACE
+ Exampes Origin: $ORIGIN
+ Examples Subdirs: $SUBDIRS
+ SDKs: $SDKS
+ CD Script Concurrency: $BEAM_CONCURRENCY"
+
+if [ -z "$DNS_NAME" ] || [ -z "$DATASTORE_NAMESPACE" ]; then
+ echo "DNS_NAME and DATASTORE_NAMESPACE substitutions must be set in the triger"
+ exit 1
+fi
+
+
+if [ "${TARGET_PR_REPO_BRANCH}" != "apache:master" ]; then
+ echo "CDLOG Merging not into the master, but into the branch apache/${TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+fi
+
+
+if [ ${PR_TYPE} != "closed" ] || [ ${MERGE_STATUS} != "true" ];
+then
+ echo "CDLOG $(date --utc '+%D %T') PR in $PR_URL for the COMMIT $PR_COMMIT is not in closed state or not merged into the Apache/Beam master repo."
+ exit 0
+fi
+
+
+echo "CDLOG Pull Request $PR_URL has been successfully merged into Apache Beam GitHub repository. Continuing the process."
+
+echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+
+apt update > /dev/null 2>&1
+
+
+apt install -y git curl > /dev/null 2>&1
+
+
+apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+
+echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+
+/etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+
+apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
Review Comment:
This will redirect only the output of second command to the `/dev/null`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1178853218
##########
playground/infrastructure/cloudbuild/playground_cd_examples.sh:
##########
@@ -0,0 +1,187 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export FORCE_CD=${FORCE_CD-'false'}
+
+# Playground FQDN
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+# GCP Datastore namespace to load examples to
+if [[ -z "${DATASTORE_NAMESPACE}" ]]; then
+ echo "DATASTORE_NAMESPACE parameter was not set. Exiting"
+ exit 1
+fi
+# Master branch commit after merge
+if [[ -z "${MERGE_COMMIT}" ]]; then
+ echo "MERGE_COMMIT paramter was not set. Exiting"
+ exit 1
+fi
+if [[ "${FORCE_CD}" != "false" && "${FORCE_CD}" != "true" ]]; then
+ echo "FORCE_CD paramter must be either 'true' or 'false'. Exiting"
+ exit 1
+fi
+
+# Constants
+export STEP=CD
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ MERGE_COMMIT=$MERGE_COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ DATASTORE_NAMESPACE=$DATASTORE_NAMESPACE
+ FORCE_CD=$FORCE_CD"
+
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+LogOutput "Installing python and dependencies."
+# set -e # Exit immediately if any command fails
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install -y --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
Review Comment:
Upgrade pip before using it to install a package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138184595
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+1. ORIGIN=$ORIGIN
+2. STEP=$STEP
+3. SUBDIRS=$SUBDIRS
+4. BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+5. PROJECT_ID=$PROJECT_ID
+6. SDK_CONFIG=$SDK_CONFIG
+7. BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+8. BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+9. SDKS=$SDKS
+10. COMMIT=$COMMIT
+11. DNS_NAME=$DNS_NAME
+12. BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+LogOutput "Installing python and dependencies"
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null 2>&1
+apt install python3.8-venv > /dev/null 2>&1
+pip install -r /workspace/beam/playground/infrastructure/requirements.txt > /dev/null 2>&1
+
+LogOutput "All packages and dependencies have been successfully installed. Starting Playground examples Deployment to https://${DNS_NAME}."
+
+# Run CD script to deploy Examples to Playground for Go, Java, Python SDK
+cd $BEAM_ROOT_DIR/playground/infrastructure
+for sdk in $SDKS
+do
+ export SERVER_ADDRESS=https://${sdk}.${DNS_NAME}
+ python3 ci_cd.py \
Review Comment:
Redirect output to the log (see CI script)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ruslan-ikhsan commented on a diff in pull request #25117: Playground ci webhook
Posted by "ruslan-ikhsan (via GitHub)" <gi...@apache.org>.
ruslan-ikhsan commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1155911005
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,154 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+if [[ -z "${PROJECT_ID}" ]]; then
+ echo "PROJECT_ID is empty or not set. Exiting"
+ exit 1
+fi
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
Review Comment:
Set now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158633875
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL;
+ PR Source Repo: $_FORK_REPO;
+ PR Branch: $_PR_BRANCH;
+ PR Commit: $_PR_COMMIT;
+ WebHook Action: $_PR_TYPE;
+ Merged to: $_TARGET_PR_REPO_BRANCH;
+ Playground DNS: $_DNS_NAME; "
+ if [[ ${_TARGET_PR_REPO_BRANCH} != "apache:master" ]]; then
+ echo "CDLOG Merging not into the master, but into the branch apache/${_TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == "closed" ]] && [[ ${_MERGE_STATUS} == "true" ]]; then
+
+ echo "CDLOG Pull Request $_PR_URL has been successfully closed and merged into Apache/Beam GitHub repository. Continuing the process."
+
+ echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git > /dev/null 2>&1
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/apache/beam/master/playground/infrastructure/cloudbuild/examples_cd.sh
Review Comment:
Also, what's the reason for fetching this file via http if we already have the repo cloned above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158556631
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
Review Comment:
We clean up environment variables when we call the script. PROJECT_ID received from command line, not from global environment variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158715258
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL;
+ PR Source Repo: $_FORK_REPO;
+ PR Branch: $_PR_BRANCH;
+ PR Commit: $_PR_COMMIT;
+ WebHook Action: $_PR_TYPE;
+ Merged to: $_TARGET_PR_REPO_BRANCH;
+ Playground DNS: $_DNS_NAME; "
+ if [[ ${_TARGET_PR_REPO_BRANCH} != "apache:master" ]]; then
+ echo "CDLOG Merging not into the master, but into the branch apache/${_TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == "closed" ]] && [[ ${_MERGE_STATUS} == "true" ]]; then
+
+ echo "CDLOG Pull Request $_PR_URL has been successfully closed and merged into Apache/Beam GitHub repository. Continuing the process."
+
+ echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git > /dev/null 2>&1
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/apache/beam/master/playground/infrastructure/cloudbuild/examples_cd.sh
Review Comment:
Yes we can use it from master for CD, since it runs from master branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158608804
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,244 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Script validates examples against runners (Java, Python, Go) built as local containers from a current branch
+#
+# Command line arguments:
+# LOG_PATH - full path lo a log file. Output is also logged to stdout (Default: /dev/null)
+# BEAM_ROOT_DIR - Path (full) to cloned repo root. Used by ci_cd.py (Default: /workspace/beam)
+# PROJECT_ID - GCP Project ID. Used by ci_cd.py (Default: test)
+# BEAM_CONCURRENCY - Number of examples to run in parallel. Used by ci_cd.py (Default: 4)
+# BEAM_VERSION - version of BEAM SDK to build containers
+# SUBDIRS - array of paths (relative to beam repo root) to search changed examples (Default: ./learning/katas ./examples ./sdks)
+# COMMIT - Git commit hash to build containers from (Default: HEAD)
+# DIFF_BASE - Git branch to compare with $COMMIT to look for changed examples (Default: origin/master)
+# SDKS - Array of SDKS to validate (Default: java python go)
+# ORIGIN - examples origin (Default: PG_EXAMPLES)
+# ALLOWLIST - List of paths (relative to the repo root) not in examples (SUBDIRS) that cause reb
+
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+
+ export "$KEY"="$VALUE"
+done
+
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export BEAM_VERSION=${BEAM_VERSION-"2.44.0"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-"4"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend"}
+
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CILOG keyword to simplify search over the global log
+ echo "CILOG $(date --utc '+%D %T') $1"
+}
+
+
+LogOutput "Input variables:
+ LOG_PATH=$LOG_PATH
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ BEAM_VERSION=$BEAM_VERSION
+ SUBDIRS=$SUBDIRS
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ ORIGIN=$ORIGIN
+ ALLOWLIST=$ALLOWLIST"
+
+# Assigning constant values
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CI
+export SDK_CONFIG="$BEAM_ROOT_DIR/playground/sdks.yaml"
+export BEAM_EXAMPLE_CATEGORIES="$BEAM_ROOT_DIR/playground/categories.yaml"
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+LogOutput "Installing python java8 and dependencies"
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+LogOutput "Installing Python environment"
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install -y --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null
+apt install -y python3.8-venv > /dev/null
+pip install -r playground/infrastructure/requirements.txt > /dev/null
+
+LogOutput "Installing JDK and Gradle"
+apt-get install openjdk-8-jdk -y > /dev/null
Review Comment:
Let's use Java 11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158645813
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL;
+ PR Source Repo: $_FORK_REPO;
+ PR Branch: $_PR_BRANCH;
+ PR Commit: $_PR_COMMIT;
+ WebHook Action: $_PR_TYPE;
+ Merged to: $_TARGET_PR_REPO_BRANCH;
+ Playground DNS: $_DNS_NAME; "
+ if [[ ${_TARGET_PR_REPO_BRANCH} != "apache:master" ]]; then
+ echo "CDLOG Merging not into the master, but into the branch apache/${_TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == "closed" ]] && [[ ${_MERGE_STATUS} == "true" ]]; then
+
+ echo "CDLOG Pull Request $_PR_URL has been successfully closed and merged into Apache/Beam GitHub repository. Continuing the process."
+
+ echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
Review Comment:
I think we can do only shallow fetch on the master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1139934839
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL;
+ PR Source Repo: $_FORK_REPO;
+ PR Branch: $_PR_BRANCH;
+ PR Commit: $_PR_COMMIT;
+ WebHook Action: $_PR_TYPE;
+ Merged to: $_TARGET_PR_REPO_BRANCH;
+ Playground DNS: $_DNS_NAME; "
+ if [[ ${_TARGET_PR_REPO_BRANCH} != "apache:master" ]]; then
+ echo "CDLOG Merge to branch apache/${_TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == "closed" ]] && [[ ${_MERGE_STATUS} == "true" ]]; then
+
+ echo "CDLOG Pull Request $_PR_URL has been successfully closed and merged into Apache/Beam GitHub repository. Continuing the process."
+
+ echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git > /dev/null 2>&1
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/akvelon/beam/master/playground/infrastructure/cloudbuild/examples_cd.sh
Review Comment:
apache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138190677
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ 1. PR URL: $_PR_URL;
+ 2. PR Source Repo: $_FORK_REPO;
+ 3. PR Branch: $_PR_BRANCH;
+ 4. PR Commit: $_PR_COMMIT;
+ 5. WebHook Action: $_PR_TYPE;
+ 6. Merged to: $_TARGET_PR_REPO_BRANCH;
+ 7. Playground DNS: $_DNS_NAME; "
+ if [[ "${_PR_BRANCH}" != "playground-ci-webhook" ]]; then
+ exit 0
+ fi
+ if [[ ${_TARGET_PR_REPO_BRANCH} == "apache:master" ]]; then
+ echo "CDLOG $(date --utc '+%D %T').
+
+ Pull Request $_PR_URL has been successfully merged into Apache/Beam Master repository (https://github.com/apache/beam).
+
+ Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git > /dev/null 2>&1
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/akvelon/beam/playground-ci-webhook/playground/infrastructure/cloudbuild/examples_cd.sh
+
+ chmod +x /tmp/examples_cd.sh
+
+ env -i bash -c "/tmp/examples_cd.sh PROJECT_ID='"'${PROJECT_ID}'"' DNS_NAME='"'${_DNS_NAME}'"' COMMIT='"'${_PR_COMMIT}'"' "
Review Comment:
--namespace parameter for ci_cd.py is also required here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138187757
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
Review Comment:
If you don't set a default value, check that parameter is not empty
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
Review Comment:
If you don't set a default value, check that parameter is not empty
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ruslan-ikhsan commented on a diff in pull request #25117: Playground ci webhook
Posted by "ruslan-ikhsan (via GitHub)" <gi...@apache.org>.
ruslan-ikhsan commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138390409
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+1. ORIGIN=$ORIGIN
+2. STEP=$STEP
+3. SUBDIRS=$SUBDIRS
+4. BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+5. PROJECT_ID=$PROJECT_ID
+6. SDK_CONFIG=$SDK_CONFIG
+7. BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+8. BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+9. SDKS=$SDKS
+10. COMMIT=$COMMIT
+11. DNS_NAME=$DNS_NAME
+12. BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+LogOutput "Installing python and dependencies"
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null 2>&1
+apt install python3.8-venv > /dev/null 2>&1
+pip install -r /workspace/beam/playground/infrastructure/requirements.txt > /dev/null 2>&1
+
+LogOutput "All packages and dependencies have been successfully installed. Starting Playground examples Deployment to https://${DNS_NAME}."
+
+# Run CD script to deploy Examples to Playground for Go, Java, Python SDK
+cd $BEAM_ROOT_DIR/playground/infrastructure
+for sdk in $SDKS
+do
+ export SERVER_ADDRESS=https://${sdk}.${DNS_NAME}
+ python3 ci_cd.py \
Review Comment:
Redirect has been enabled
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,243 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Script validates examples against runners (Java, Python, Go) built as local containers from a current branch
+#
+# Command line arguments:
+# LOG_PATH - full path lo a log file. Output is also logged to stdout (Default: /dev/null)
+# BEAM_ROOT_DIR - Path (full) to cloned repo root. Used by ci_cd.py (Default: /workspace/beam)
+# PROJECT_ID - GCP Project ID. Used by ci_cd.py (Default: test)
+# BEAM_CONCURRENCY - Number of examples to run in parallel. Used by ci_cd.py (Default: 4)
+# BEAM_VERSION - version of BEAM SDK to build containers
+# SUBDIRS - array of paths (relative to beam repo root) to search changed examples (Default: ./learning/katas ./examples ./sdks)
+# COMMIT - Git commit hash to build containers from (Default: HEAD)
+# DIFF_BASE - Git branch to compare with $COMMIT to look for changed examples (Default: origin/master)
+# SDKS - Array of SDKS to validate (Default: java python go)
+# ORIGIN - examples origin (Default: PG_EXAMPLES)
+# ALLOWLIST - List of paths (relative to the repo root) not in examples (SUBDIRS) that cause reb
+
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+
+ export "$KEY"="$VALUE"
+done
+
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID-"test"}
+export BEAM_VERSION=${BEAM_VERSION-"2.44.0"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-"4"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend"}
+
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CILOG keyword to simplify search over the global log
+ echo "CILOG $(date --utc '+%D %T') $1"
+}
+
+
+LogOutput "Input variables:
+ LOG_PATH=$LOG_PATH
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ BEAM_VERSION=$BEAM_VERSION
+ SUBDIRS=$SUBDIRS
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ ORIGIN=$ORIGIN
+ ALLOWLIST=$ALLOWLIST"
+
+# Assigning constant values
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+export STEP=CI
+export SDK_CONFIG="$BEAM_ROOT_DIR/playground/sdks.yaml"
+export BEAM_EXAMPLE_CATEGORIES="$BEAM_ROOT_DIR/playground/categories.yaml"
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+LogOutput "Installing python java8 and dependencies"
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+LogOutput "Installing Python environment"
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
Review Comment:
Added
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+1. ORIGIN=$ORIGIN
+2. STEP=$STEP
+3. SUBDIRS=$SUBDIRS
+4. BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+5. PROJECT_ID=$PROJECT_ID
+6. SDK_CONFIG=$SDK_CONFIG
+7. BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+8. BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+9. SDKS=$SDKS
+10. COMMIT=$COMMIT
+11. DNS_NAME=$DNS_NAME
+12. BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+LogOutput "Installing python and dependencies"
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install --reinstall python3.8-distutils > /dev/null 2>&1
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ruslan-ikhsan commented on a diff in pull request #25117: Playground ci webhook
Posted by "ruslan-ikhsan (via GitHub)" <gi...@apache.org>.
ruslan-ikhsan commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138388015
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ 1. PR URL: $_PR_URL;
+ 2. PR Source Repo: $_FORK_REPO;
+ 3. PR Branch: $_PR_BRANCH;
+ 4. PR Commit: $_PR_COMMIT;
+ 5. WebHook Action: $_PR_TYPE;
+ 6. Merged to: $_TARGET_PR_REPO_BRANCH;
+ 7. Playground DNS: $_DNS_NAME; "
+ if [[ "${_PR_BRANCH}" != "playground-ci-webhook" ]]; then
+ exit 0
+ fi
+ if [[ ${_TARGET_PR_REPO_BRANCH} == "apache:master" ]]; then
Review Comment:
@rshamunov reversed, as in CI script
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1139925305
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,154 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+if [[ -z "${PROJECT_ID}" ]]; then
+ echo "PROJECT_ID is empty or not set. Exiting"
+ exit 1
+fi
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=$STEP
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+LogOutput "Installing python and dependencies"
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install -y --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null 2>&1
+apt install -y python3.8-venv > /dev/null 2>&1
+pip install -r /workspace/beam/playground/infrastructure/requirements.txt > /dev/null 2>&1
+
+LogOutput "All packages and dependencies have been successfully installed. Starting Playground examples Deployment to https://${DNS_NAME}."
+
+LogOutput "Checking changed files in the PR"
+
+diff=($(git diff --name-only $DIFF_BASE...$SOURCE_BRANCH | tr '\n' ' '))
+
+for file in "${diff[@]}"; do
+ if echo $file == *learning/katas/ || echo $file == *examples/* || echo $file == *sdks/*
Review Comment:
You also need to check if the change is related to an example with tags
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1130816196
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,243 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Script validates examples against runners (Java, Python, Go) built as local containers from a current branch
+#
+# Command line arguments:
+# LOG_PATH - full path lo a log file. Output is also logged to stdout (Default: /dev/null)
+# BEAM_ROOT_DIR - Path (full) to cloned repo root. Used by ci_cd.py (Default: /workspace/beam)
+# PROJECT_ID - GCP Project ID. Used by ci_cd.py (Default: test)
+# BEAM_CONCURRENCY - Number of examples to run in parallel. Used by ci_cd.py (Default: 4)
+# BEAM_VERSION - version of BEAM SDK to build containers
+# SUBDIRS - array of paths (relative to beam repo root) to search changed examples (Default: ./learning/katas ./examples ./sdks)
+# COMMIT - Git commit hash to build containers from (Default: HEAD)
+# DIFF_BASE - Git branch to compare with $COMMIT to look for changed examples (Default: origin/master)
+# SDKS - Array of SDKS to validate (Default: java python go)
+# ORIGIN - examples origin (Default: PG_EXAMPLES)
+# ALLOWLIST - List of paths (relative to the repo root) not in examples (SUBDIRS) that cause reb
+
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+
+ export "$KEY"="$VALUE"
+done
+
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID-"test"}
+export BEAM_VERSION=${BEAM_VERSION-"2.44.0"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-"4"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend"}
+
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CILOG keyword to simplify search over the global log
+ echo "CILOG $(date --utc '+%D %T') $1"
+}
+
+
+LogOutput "Input variables:
+ LOG_PATH=$LOG_PATH
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ BEAM_VERSION=$BEAM_VERSION
+ SUBDIRS=$SUBDIRS
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ ORIGIN=$ORIGIN
+ ALLOWLIST=$ALLOWLIST"
+
+# Assigning constant values
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+export STEP=CI
+export SDK_CONFIG="$BEAM_ROOT_DIR/playground/sdks.yaml"
+export BEAM_EXAMPLE_CATEGORIES="$BEAM_ROOT_DIR/playground/categories.yaml"
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+LogOutput "Installing python java8 and dependencies"
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+LogOutput "Installing Python environment"
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
Review Comment:
add '-y' (yes) parameter to all apt commands if it's applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1175149886
##########
playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/terraform.tf:
##########
@@ -22,7 +22,7 @@ terraform {
required_providers {
google = {
source = "hashicorp/google"
- version = "~> 4.40.0"
+ version = "~> 4.62.0"
Review Comment:
Do we want to update version in other TF scripts too? (not in this PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1178853983
##########
playground/infrastructure/cloudbuild/playground_cd_manual.yaml:
##########
@@ -0,0 +1,102 @@
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL
+ Merged to: $_TARGET_PR_REPO_BRANCH
+ WebHook Action: $_PR_TYPE
+ Merge Status: $_MERGE_STATUS
+ Merge commit: $_MERGE_COMMIT
+ Playground DNS: $_DNS_NAME
+ Datastore namespace: $_DATASTORE_NAMESPACE
+ Exampes Origin: $_ORIGIN
+ Examples Subdirs: $_SUBDIRS
+ SDKs: $_SDKS
+ CD Script Concurrency: $_BEAM_CONCURRENCY"
+
+ if [[ -z "$_DNS_NAME" || -z "$_DATASTORE_NAMESPACE" ]]; then
+ echo "_DNS_NAME and _DATASTORE_NAMESPACE substitutions must be set in the triger"
+ exit 1
+ fi
+
+
+ if [[ ${_TARGET_PR_REPO_BRANCH} != "apache:master" ]]; then
+ echo "CDLOG Merging not into the master, but into the branch apache/${_TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+ fi
+
+
+ if [[ ${_PR_TYPE} != "closed" ]] || [[ ${_MERGE_STATUS} != "true" ]];
+ then
+ echo "CDLOG $(date --utc '+%D %T') PR in $_PR_URL for the COMMIT $_PR_COMMIT is not in closed state or not merged into the Apache/Beam master repo."
+ exit 0
+ fi
+
+
+ echo "CDLOG Pull Request $_PR_URL has been successfully merged into
+ Apache Beam GitHub repository. Continuing the process."
+
+ echo "CDLOG Continous Deployment of Playground Examples (CD) in the
+ progress."
+
+
+ apt update > /dev/null 2>&1
+
+
+ apt install -y git curl > /dev/null 2>&1
+
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null
+ 2>&1
+
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" >
+ /dev/null 2>&1 | tee -a
+
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null
+ 2>&1 | apt-key add - > /dev/null 2>&1
Review Comment:
`apt-key` is deprecated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1175429772
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_deploy.sh:
##########
@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
Review Comment:
Add License to all scripts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158606913
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"learning/katas sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CD
+
+LogOutput "Installing python and dependencies."
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install -y --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null 2>&1
+apt install -y python3.8-venv > /dev/null 2>&1
Review Comment:
what's the point in installing `python3.8-venv` if we are not using it after and installing pip packages globally anyway?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158689481
##########
playground/infrastructure/cloudbuild/cloudbuild_helm_update.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk > /dev/null
+
+ git clone --branch merged_pg_gke_deployment
+ https://github.com/akvelon/beam.git
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ cd beam
+
+ mkdir playground/terraform/environment/myenv
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/terraform.tfvars
+ playground/terraform/environment/myenv/terraform.tfvars
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/state.tfbackend
+ playground/terraform/environment/myenv/state.tfbackend
+
+ cat playground/terraform/environment/myenv/terraform.tfvars
+
+ cat playground/terraform/environment/myenv/state.tfbackend
+
+ gcloud auth configure-docker us-east1-docker.pkg.dev
+
+ gcloud container clusters get-credentials --region us-east1-b
+ playground-gke --project cloudbuildtest-382210
+
+ ./gradlew playground:terraform:gkebackend -Pproject_environment=myenv
+ -Pdocker-tag=$_TAG_NAME -Pdns-name=$_DNS_NAME -Psdk-tag=$_SDK_TAG
Review Comment:
It is simple as is. We have dediacted shell files for more complex cases (CI/CD)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1104590098
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,165 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Added set -x to show output into cloud build console to the bucket
+set -x
+
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+#Install python java8 and dependencies
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+# Env configuration commands
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null
+apt install python3.8-venv > /dev/null
+pip install -r playground/infrastructure/requirements.txt > /dev/null
+
+# Install jdk and gradle
+apt-get install openjdk-8-jdk -y > /dev/null
+curl -L https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip -o gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+unzip gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+export PATH=$PATH:gradle-${GRADLE_VERSION}/bin > /dev/null
+
+# Install go
+curl -OL https://golang.org/dl/go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+tar -C /usr/local -xvf go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+export PATH=$PATH:/usr/local/go/bin > /dev/null
+
+# Install Docker
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - > /dev/null
+add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+apt update > /dev/null && apt install -y docker-ce > /dev/null
+
+# Assigning required values for CI_CD.py script
+export \
+ORIGIN=PG_EXAMPLES \
+STEP=CI \
+SUBDIRS="./learning/katas ./examples ./sdks" \
+GOOGLE_CLOUD_PROJECT=${PROJECT_ID} \
+BEAM_ROOT_DIR="../.." \
+SDK_CONFIG="../sdks.yaml" \
+BEAM_EXAMPLE_CATEGORIES="../categories.yaml" \
+BEAM_CONCURRENCY=4 \
+BEAM_VERSION=2.43.0 \
Review Comment:
Replace to an optional parameter for a script with default value "2.43.0". So we don't need to modify the script to update to higher version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1104647432
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,165 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Added set -x to show output into cloud build console to the bucket
+set -x
+
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+#Install python java8 and dependencies
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+# Env configuration commands
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null
+apt install python3.8-venv > /dev/null
+pip install -r playground/infrastructure/requirements.txt > /dev/null
+
+# Install jdk and gradle
+apt-get install openjdk-8-jdk -y > /dev/null
+curl -L https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip -o gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+unzip gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+export PATH=$PATH:gradle-${GRADLE_VERSION}/bin > /dev/null
+
+# Install go
+curl -OL https://golang.org/dl/go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+tar -C /usr/local -xvf go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+export PATH=$PATH:/usr/local/go/bin > /dev/null
+
+# Install Docker
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - > /dev/null
+add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+apt update > /dev/null && apt install -y docker-ce > /dev/null
+
+# Assigning required values for CI_CD.py script
+export \
+ORIGIN=PG_EXAMPLES \
+STEP=CI \
+SUBDIRS="./learning/katas ./examples ./sdks" \
+GOOGLE_CLOUD_PROJECT=${PROJECT_ID} \
+BEAM_ROOT_DIR="../.." \
+SDK_CONFIG="../sdks.yaml" \
+BEAM_EXAMPLE_CATEGORIES="../categories.yaml" \
+BEAM_CONCURRENCY=4 \
+BEAM_VERSION=2.43.0 \
+sdks=("java" "python" "go") \
+allowlist=("playground/infrastructure" "playground/backend")
+
+# Check whether commit is tagged
+tag_name=$(git tag --points-at $commit_sha)
+
+# Get diff
+if [ -z $base_ref ] || [ $base_ref == "master" ]
+then
+ base_ref=origin/master
+fi
+diff=$(git diff --name-only $base_ref $commit_sha | tr '\n' ' ')
+
+# Check if there are Examples as
+for sdk in "${sdks[@]}"
+do
+ cd playground/infrastructure
+ python3 checker.py \
+ --verbose \
+ --sdk SDK_"${sdk^^}" \
+ --allowlist "${allowlist[@]}" \
+ --paths ${diff}
+ checker_status=$?
+ cd -
+ if [ $checker_status -eq 0 ]
+ then
+ echo "Checker has found changed examples for ${sdk^^}" >> ${1}
+ example_has_changed=True
+ elif [ $checker_status -eq 11 ]
+ then
+ echo "Checker has not found changed examples for ${sdk^^}" >> ${1}
+ example_has_changed=False
+ else
+ echo "Error: Checker is broken" >> ${1}
+ exit 1
+ fi
+
+# Run main logic if examples have been changed
+ if [[ $example_has_changed == "True" ]]
+ then
+ if [ -z "${tag_name}" ] && [ "${commit_sha}" ]
Review Comment:
Re-work condition:
if [ "${tag_name}" ] DOCKERTAG=${tag_name}
elif [ "${commit_sha}" ] DOCKERTAG=${commit_sha}
else error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1178854864
##########
playground/infrastructure/cloudbuild/playground_ci_examples.sh:
##########
@@ -0,0 +1,246 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Script validates examples against runners (Java, Python, Go) built as local containers from a current branch
+#
+# Command line arguments:
+# LOG_PATH - full path lo a log file. Output is also logged to stdout (Default: /dev/null)
+# BEAM_ROOT_DIR - Path (full) to cloned repo root. Used by ci_cd.py (Default: /workspace/beam)
+# PROJECT_ID - GCP Project ID. Used by ci_cd.py (Default: test)
+# BEAM_CONCURRENCY - Number of examples to run in parallel. Used by ci_cd.py (Default: 4)
+# BEAM_VERSION - version of BEAM SDK to build containers
+# SUBDIRS - array of paths (relative to beam repo root) to search changed examples (Default: ./learning/katas ./examples ./sdks)
+# COMMIT - Git commit hash to build containers from (Default: HEAD)
+# DIFF_BASE - Git branch to compare with $COMMIT to look for changed examples (Default: origin/master)
+# SDKS - Array of SDKS to validate (Default: java python go)
+# ORIGIN - examples origin (Default: PG_EXAMPLES)
+# ALLOWLIST - List of paths (relative to the repo root) not in examples (SUBDIRS) that cause reb
+
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+
+ export "$KEY"="$VALUE"
+done
+
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID-"test"}
+export BEAM_VERSION=${BEAM_VERSION-"2.44.0"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-"4"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend"}
+
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CILOG keyword to simplify search over the global log
+ echo "CILOG $(date --utc '+%D %T') $1"
+}
+
+
+LogOutput "Input variables:
+ LOG_PATH=$LOG_PATH
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ BEAM_VERSION=$BEAM_VERSION
+ SUBDIRS=$SUBDIRS
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ ORIGIN=$ORIGIN
+ ALLOWLIST=$ALLOWLIST"
+
+# Assigning constant values
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+export STEP=CI
+export SDK_CONFIG="$BEAM_ROOT_DIR/playground/sdks.yaml"
+export BEAM_EXAMPLE_CATEGORIES="$BEAM_ROOT_DIR/playground/categories.yaml"
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+LogOutput "Installing python java8 and dependencies"
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+LogOutput "Installing Python environment"
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
Review Comment:
Again, there is no reason to use non-system version of python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1139942865
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,154 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+if [[ -z "${PROJECT_ID}" ]]; then
+ echo "PROJECT_ID is empty or not set. Exiting"
+ exit 1
+fi
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
Review Comment:
You didn't set SOURCE_BRANCH in the trigger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ruslan-ikhsan commented on a diff in pull request #25117: Playground ci webhook
Posted by "ruslan-ikhsan (via GitHub)" <gi...@apache.org>.
ruslan-ikhsan commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138389789
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
Review Comment:
Check if variable is not empty condition added
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158550476
##########
playground/infrastructure/cloudbuild/cloudbuild_helm_update.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk > /dev/null
+
+ git clone --branch merged_pg_gke_deployment
+ https://github.com/akvelon/beam.git
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ cd beam
+
+ mkdir playground/terraform/environment/myenv
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/terraform.tfvars
+ playground/terraform/environment/myenv/terraform.tfvars
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/state.tfbackend
+ playground/terraform/environment/myenv/state.tfbackend
+
+ cat playground/terraform/environment/myenv/terraform.tfvars
+
+ cat playground/terraform/environment/myenv/state.tfbackend
+
+ gcloud auth configure-docker us-east1-docker.pkg.dev
+
+ gcloud container clusters get-credentials --region us-east1-b
+ playground-gke --project cloudbuildtest-382210
+
+ ./gradlew playground:terraform:gkebackend -Pproject_environment=myenv
+ -Pdocker-tag=$_TAG_NAME -Pdns-name=$_DNS_NAME -Psdk-tag=$_SDK_TAG
+ -Pdocker-repository-root=$_ARTIFACT_REGISTRY_NAME
+ -Pdatastore-namespace=$_DATASTORE_NAMESPACE
+
+ cat playground/infrastructure/helm-playground/values.yaml
+ entrypoint: bash
+timeout: 7200s
+logsBucket: 'gs://$_PLAYGROUND_TFSTATE_BUCKET'
+options:
+ machineType: E2_HIGHCPU_32
+ logging: GCS_ONLY
+substitutions:
+ _PLAYGROUND_TFSTATE_BUCKET: tfstate-pg-ruslan1
Review Comment:
Give it neuatrl name: 'tfstage-pg-cloudbuild-update-environment'
##########
playground/infrastructure/cloudbuild/cloudbuild_helm_update.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk > /dev/null
+
+ git clone --branch merged_pg_gke_deployment
+ https://github.com/akvelon/beam.git
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ cd beam
+
+ mkdir playground/terraform/environment/myenv
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/terraform.tfvars
+ playground/terraform/environment/myenv/terraform.tfvars
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/state.tfbackend
+ playground/terraform/environment/myenv/state.tfbackend
+
+ cat playground/terraform/environment/myenv/terraform.tfvars
+
+ cat playground/terraform/environment/myenv/state.tfbackend
+
+ gcloud auth configure-docker us-east1-docker.pkg.dev
+
+ gcloud container clusters get-credentials --region us-east1-b
+ playground-gke --project cloudbuildtest-382210
+
+ ./gradlew playground:terraform:gkebackend -Pproject_environment=myenv
+ -Pdocker-tag=$_TAG_NAME -Pdns-name=$_DNS_NAME -Psdk-tag=$_SDK_TAG
+ -Pdocker-repository-root=$_ARTIFACT_REGISTRY_NAME
+ -Pdatastore-namespace=$_DATASTORE_NAMESPACE
+
+ cat playground/infrastructure/helm-playground/values.yaml
+ entrypoint: bash
+timeout: 7200s
+logsBucket: 'gs://$_PLAYGROUND_TFSTATE_BUCKET'
+options:
+ machineType: E2_HIGHCPU_32
+ logging: GCS_ONLY
+substitutions:
+ _PLAYGROUND_TFSTATE_BUCKET: tfstate-pg-ruslan1
Review Comment:
Give it neuatral name: 'tfstage-pg-cloudbuild-update-environment'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158626426
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"learning/katas sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CD
+
+LogOutput "Installing python and dependencies."
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install -y --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
Review Comment:
run the pip upgrade before using it to install the `google-api-python-client`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158697265
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"learning/katas sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CD
+
+LogOutput "Installing python and dependencies."
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
Review Comment:
It produces lots of junk to output
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158704423
##########
playground/infrastructure/cloudbuild/playground_ci_stable.yaml:
##########
@@ -0,0 +1,112 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ echo "Search for CILOG keyword to find valuable logs entries"
+
+ echo "CILOG $(date --utc '+%D %T') Trigger run inputs: PR URL:
+ $_PR_URL; PR Source Repo: $_FORK_REPO; PR Branch: $_PR_BRANCH; PR
+ Commit: $_PR_COMMIT; WebHook Action: $_PR_TYPE; Merge to: $_BASE_REF;"
+
+ if [[ "${_BASE_REF}" != "master" ]]; then
+ echo "CILOG Merging not into master, but into the branch apache/${_BASE_REF}. Exiting."
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == @(opened|synchronize) ]]; then
+
+ echo "CILOG See also public logs: ${_PUBLIC_LOG_URL}"
+
+ echo "CILOG $(date --utc '+%D %T') Examples validation (CI) has started"
+
+ echo "$(date --utc '+%D %T') Examples validation (CI) has started" >> ${_PUBLIC_LOG_LOCAL}
+
+ apt update > /dev/null
+
+ apt-get install -y git curl apt-transport-https ca-certificates gnupg > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null
+
+ git clone --branch master https://github.com/apache/beam.git
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/apache/beam/master/playground/infrastructure/cloudbuild/examples_ci.sh
Review Comment:
We don't want to allow to use this script from a PR that we validate. It should be downloaded from trusted source (master branch).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158652589
##########
playground/infrastructure/cloudbuild/playground_ci_stable.yaml:
##########
@@ -0,0 +1,112 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ echo "Search for CILOG keyword to find valuable logs entries"
+
+ echo "CILOG $(date --utc '+%D %T') Trigger run inputs: PR URL:
+ $_PR_URL; PR Source Repo: $_FORK_REPO; PR Branch: $_PR_BRANCH; PR
+ Commit: $_PR_COMMIT; WebHook Action: $_PR_TYPE; Merge to: $_BASE_REF;"
+
+ if [[ "${_BASE_REF}" != "master" ]]; then
+ echo "CILOG Merging not into master, but into the branch apache/${_BASE_REF}. Exiting."
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == @(opened|synchronize) ]]; then
+
+ echo "CILOG See also public logs: ${_PUBLIC_LOG_URL}"
+
+ echo "CILOG $(date --utc '+%D %T') Examples validation (CI) has started"
+
+ echo "$(date --utc '+%D %T') Examples validation (CI) has started" >> ${_PUBLIC_LOG_LOCAL}
+
+ apt update > /dev/null
+
+ apt-get install -y git curl apt-transport-https ca-certificates gnupg > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null
+
+ git clone --branch master https://github.com/apache/beam.git
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/apache/beam/master/playground/infrastructure/cloudbuild/examples_ci.sh
Review Comment:
Though, this requires come careful considerations, do we expect to be able to easily verify cloudbuild changes or only after they are merged into the master?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158647591
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL;
+ PR Source Repo: $_FORK_REPO;
+ PR Branch: $_PR_BRANCH;
+ PR Commit: $_PR_COMMIT;
+ WebHook Action: $_PR_TYPE;
+ Merged to: $_TARGET_PR_REPO_BRANCH;
+ Playground DNS: $_DNS_NAME; "
+ if [[ ${_TARGET_PR_REPO_BRANCH} != "apache:master" ]]; then
+ echo "CDLOG Merging not into the master, but into the branch apache/${_TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == "closed" ]] && [[ ${_MERGE_STATUS} == "true" ]]; then
+
+ echo "CDLOG Pull Request $_PR_URL has been successfully closed and merged into Apache/Beam GitHub repository. Continuing the process."
+
+ echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
Review Comment:
and on the branch from which we are bulding also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1175147460
##########
playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/iam.tf:
##########
@@ -84,3 +84,13 @@ resource "google_project_iam_member" "playground_cicd_sa_roles" {
member = "serviceAccount:${google_service_account.pg_cloudbuild_cicd_runner_sa.email}"
Review Comment:
We need dedicated accounts for CI and for CD triggers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1130815867
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+1. ORIGIN=$ORIGIN
+2. STEP=$STEP
+3. SUBDIRS=$SUBDIRS
+4. BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+5. PROJECT_ID=$PROJECT_ID
+6. SDK_CONFIG=$SDK_CONFIG
+7. BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+8. BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+9. SDKS=$SDKS
+10. COMMIT=$COMMIT
+11. DNS_NAME=$DNS_NAME
+12. BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+LogOutput "Installing python and dependencies"
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install --reinstall python3.8-distutils > /dev/null 2>&1
Review Comment:
add '-y' (yes) parameter to all apt commands if it's applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1139903494
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL;
+ PR Source Repo: $_FORK_REPO;
Review Comment:
Why do you need fork if the change merged to master?
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL;
+ PR Source Repo: $_FORK_REPO;
+ PR Branch: $_PR_BRANCH;
+ PR Commit: $_PR_COMMIT;
+ WebHook Action: $_PR_TYPE;
+ Merged to: $_TARGET_PR_REPO_BRANCH;
+ Playground DNS: $_DNS_NAME; "
+ if [[ ${_TARGET_PR_REPO_BRANCH} != "apache:master" ]]; then
+ echo "CDLOG Merge to branch apache/${_TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == "closed" ]] && [[ ${_MERGE_STATUS} == "true" ]]; then
+
+ echo "CDLOG Pull Request $_PR_URL has been successfully closed and merged into Apache/Beam GitHub repository. Continuing the process."
+
+ echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git > /dev/null 2>&1
Review Comment:
Do you need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on PR #25117:
URL: https://github.com/apache/beam/pull/25117#issuecomment-1546267087
Replaced by #26513
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1175147460
##########
playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/iam.tf:
##########
@@ -84,3 +84,13 @@ resource "google_project_iam_member" "playground_cicd_sa_roles" {
member = "serviceAccount:${google_service_account.pg_cloudbuild_cicd_runner_sa.email}"
Review Comment:
We need dedicated role for CI and for CD triggers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158596850
##########
playground/infrastructure/cloudbuild/cloudbuild_helm_update.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk > /dev/null
+
+ git clone --branch merged_pg_gke_deployment
+ https://github.com/akvelon/beam.git
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ cd beam
+
+ mkdir playground/terraform/environment/myenv
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/terraform.tfvars
+ playground/terraform/environment/myenv/terraform.tfvars
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/state.tfbackend
+ playground/terraform/environment/myenv/state.tfbackend
+
+ cat playground/terraform/environment/myenv/terraform.tfvars
+
+ cat playground/terraform/environment/myenv/state.tfbackend
+
+ gcloud auth configure-docker us-east1-docker.pkg.dev
+
+ gcloud container clusters get-credentials --region us-east1-b
+ playground-gke --project cloudbuildtest-382210
+
+ ./gradlew playground:terraform:gkebackend -Pproject_environment=myenv
+ -Pdocker-tag=$_TAG_NAME -Pdns-name=$_DNS_NAME -Psdk-tag=$_SDK_TAG
Review Comment:
Can we extract this into a `.sh` file so it would be easier to understand?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158465568
##########
playground/infrastructure/cloudbuild/playground_deploy_cloudbuild_part_1.yaml:
##########
@@ -0,0 +1,111 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt-get update >/dev/null && apt-get install -y git build-essential
+ unzip software-properties-common ca-certificates curl wget gnupg
+ lsb-release openjdk-11-jdk >/dev/null
+
+ curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
Review Comment:
why `focal`? Please use the actually used version of the system when adding a repo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158713468
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"learning/katas sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CD
+
+LogOutput "Installing python and dependencies."
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install -y --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null 2>&1
+apt install -y python3.8-venv > /dev/null 2>&1
+pip install -r /workspace/beam/playground/infrastructure/requirements.txt > /dev/null 2>&1
+
+LogOutput "Python and dependencies have been successfully installed."
+
+LogOutput "Checking what files were changed in the PR."
+
+git fetch --all > /dev/null 2>&1
+
+diff_log=$(git diff --name-only $DIFF_BASE...forked/$SOURCE_BRANCH)
+diff=($(echo "$diff_log" | tr '\n' ' '))
+LogOutput "List of changed files in PR $SOURCE_BRANCH merging into $DIFF_BASE are:
+$diff_log"
+
+LogOutput "Looking for changes that require CD validation for [$SDKS] SDKs"
+allowlist_array=($ALLOWLIST)
+for sdk in $SDKS
+do
+ eval "example_for_${sdk}_changed"='False'
+ LogOutput "------------------Starting checker.py for SDK_${sdk^^}------------------"
+ cd $BEAM_ROOT_DIR/playground/infrastructure
+ python3 checker.py \
+ --verbose \
+ --sdk SDK_"${sdk^^}" \
+ --allowlist "" \
+ --paths "${diff[@]}"
+ checker_status=$?
+ if [ $checker_status -eq 0 ]; then
+ LogOutput "Checker found changed examples for SDK_${sdk^^}"
+ eval "example_for_${sdk}_changed"='True'
+ elif [ $checker_status -eq 11 ]; then
+ LogOutput "Checker did not find any changed examples for SDK_${sdk^^}"
+ eval "example_for_${sdk}_changed"='False'
+ continue
+ else
+ LogOutput "Error: Checker is broken. Exiting the script."
+ exit 1
+ fi
+
+ result=$(eval echo '$'"example_for_${sdk}_changed")
+
+ if [[ $result == True ]]; then
+ LogOutput "Running ci_cd.py for SDK $sdk"
+
+ export SERVER_ADDRESS=https://${sdk}.${DNS_NAME}
+ python3 ci_cd.py \
Review Comment:
Let's use global Python. No need to create venvs for temp machines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158617540
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL;
+ PR Source Repo: $_FORK_REPO;
+ PR Branch: $_PR_BRANCH;
+ PR Commit: $_PR_COMMIT;
+ WebHook Action: $_PR_TYPE;
+ Merged to: $_TARGET_PR_REPO_BRANCH;
+ Playground DNS: $_DNS_NAME; "
+ if [[ ${_TARGET_PR_REPO_BRANCH} != "apache:master" ]]; then
+ echo "CDLOG Merging not into the master, but into the branch apache/${_TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == "closed" ]] && [[ ${_MERGE_STATUS} == "true" ]]; then
+
+ echo "CDLOG Pull Request $_PR_URL has been successfully closed and merged into Apache/Beam GitHub repository. Continuing the process."
+
+ echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git > /dev/null 2>&1
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/apache/beam/master/playground/infrastructure/cloudbuild/examples_cd.sh
Review Comment:
we should fetch this file from the commit from which we are running the build. Otherwise this can lead to very hard to debug problems
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158644282
##########
playground/terraform/infrastructure/cloudbuild-manual-setup/01.setup/variables.tf:
##########
@@ -16,12 +16,20 @@
# under the License.
variable "project_id" {
- type = string
description = "The ID of the Google Cloud project within which resources are provisioned"
}
-variable "cloudbuild_service_account_id" {
- type = string
- description = "The ID of the cloud build service account responsible for provisioning Google Cloud resources"
- default = "playground-cloudbuild-sa"
+variable "pg_cloudbuild_deployer_sa_name" {
+ description = "The ID of the cloud build service account responsible for deploying the Playground"
+ default = "pg-cloudbuild-deployer-sa"
}
+
+variable "pg_cloudbuild_helm_updater_sa_name" {
+ description = "The ID of the cloud build service account responsible for updating the Helm"
+ default = "pg-cloudbuild-updater-sa"
+}
+
+variable "playground_cloudbuild_updater_sa_name" {
+ description = "The ID of the cloud build service account responsible for running CI/CD checks and scripts"
+ default = "pg-cloudbuild-cicd-runner-sa"
Review Comment:
CI and CD completely different triggers with different permission requirements. We need a dedicated trigger for CI and for CD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1178846235
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_deploy.sh:
##########
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#!/usr/bin/env bash
+export DEBIAN_FRONTEND=noninteractive
+apt-get -qq update
+apt-get -qq install -y wget unzip software-properties-common git curl apt-transport-https ca-certificates gnupg jq lsb-release
+
+echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
+curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | tee /usr/share/keyrings/cloud.google.gpg
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
+echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" |tee /etc/apt/sources.list.d/docker.list
+
+wget -nv https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+unzip terraform_1.4.2_linux_amd64.zip
+mv terraform /usr/local/bin/terraform
+
+curl -fsSLo get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
+chmod +x get_helm.sh && ./get_helm.sh
+
+apt-get -qq update
+apt-get -qq install -y google-cloud-sdk-gke-gcloud-auth-plugin google-cloud-sdk openjdk-11-jdk kubectl docker-ce
+
+git clone --branch $BRANCH_NAME $REPO_NAME --single-branch
+cd beam
+mkdir playground/terraform/environment/$ENVIRONMENT_NAME
+echo "---- ENV OUTPUT---"
+env | grep TF_VAR
+echo "bucket = \"${TF_VAR_state_bucket}\"" > playground/terraform/environment/$ENVIRONMENT_NAME/state.tfbackend
+echo "---- FILE OUTPUT ---"
+cat playground/terraform/environment/$ENVIRONMENT_NAME/state.tfbackend
+
+gcloud auth configure-docker $TF_VAR_region-docker.pkg.dev
+gcloud container clusters get-credentials --region $TF_VAR_zone $TF_VAR_gke_name --project $TF_VAR_project_id
+
+./gradlew playground:terraform:gkebackend -Pproject_id=$TF_VAR_project_id -Pproject_environment=$ENVIRONMENT_NAME -Pdocker-tag=$CONTAINER_TAG -Pdns-name=$DNS_NAME -Psdk-tag=$SDK_TAG -Pdocker-repository-root=$TF_VAR_region-docker.pkg.dev/$TF_VAR_project_id/$TF_VAR_repository_id -Pdatastore-namespace=$DATASTORE_NAMESPACE
Review Comment:
Add empty line to the end of file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1178844235
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_ci_examples.sh:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#!/usr/bin/env bash
+echo "Search for CILOG keyword to find valuable logs entries"
+echo "CILOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL:$_PR_URL
+ PR Source Repo: $_FORK_REPO
+ PR Branch: $_PR_BRANCH
+ PRCommit: $_PR_COMMIT
+ WebHook Action: $_PR_TYPE
+ Merge to: $_BASE_REF"
+
+
+if [[ "${_BASE_REF}" != "master" ]]; then
+ echo "CILOG Merge to branch apache/${_BASE_REF}. Exiting"
+ exit 0
+fi
+
+if [[ ${_PR_TYPE} == @(opened|synchronize) ]]; then
+
+ echo "CILOG See also public logs: ${_PUBLIC_LOG_URL}"
+
+ echo "CILOG $(date --utc '+%D %T') Examples validation (CI) has started"
+
+ echo "$(date --utc '+%D %T') Examples validation (CI) has started" >> ${_PUBLIC_LOG_LOCAL}
+
+ apt update > /dev/null
+
+ apt-get install -y git curl apt-transport-https ca-certificates gnupg > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null
+
+ curl -X POST -H "Accept: application/vnd.github+json" -H "Authorization: Bearer $${PAT}" -H "X-GitHub-Api-Version: 2022-11-28" https://api.github.com/repos/apache/beam/statuses/${_PR_COMMIT} -d '{"state":"pending","target_url":null,"description":"Examples validation (CI) for current commit is in progress","context":"GCP Cloud Build CI/CD"}'
+
+ echo "CILOG $(date --utc '+%D %T') Starting validation script"
+
+ echo "$(date --utc '+%D %T') Starting validation script" >> ${_PUBLIC_LOG_LOCAL}
+
+ git clone --branch master https://github.com/apache/beam.git
+
+ git remote add forked https://github.com/${_FORK_REPO}.git
+
+ chmod +x ${_CI_SCRIPT_PATH}
+
+ env -i bash -c "${_CI_SCRIPT_PATH} PROJECT_ID='"'${PROJECT_ID}'"' LOG_PATH='"'${_PUBLIC_LOG_LOCAL}'"' BEAM_VERSION='"'${_BEAM_VERSION}'"' COMMIT='"'${_PR_COMMIT}'"' BEAM_CONCURRENCY='"'${_BEAM_CONCURRENCY}'"'"
Review Comment:
Again, what's the point of using `bash -c`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1178838770
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_cd_examples.sh:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#!/usr/bin/env bash
+echo "Search for CDLOG keyword to find valuable logs entries"
+echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $PR_URL
+ Merged to: $TARGET_PR_REPO_BRANCH
+ WebHook Action: $PR_TYPE
+ Merge Status: $MERGE_STATUS
+ Merge commit: $MERGE_COMMIT
+ Playground DNS: $DNS_NAME
+ Datastore namespace: $DATASTORE_NAMESPACE
+ Exampes Origin: $ORIGIN
+ Examples Subdirs: $SUBDIRS
+ SDKs: $SDKS
+ CD Script Concurrency: $BEAM_CONCURRENCY"
+
+if [ -z "$DNS_NAME" ] || [ -z "$DATASTORE_NAMESPACE" ]; then
+ echo "DNS_NAME and DATASTORE_NAMESPACE substitutions must be set in the triger"
+ exit 1
+fi
+
+
+if [ "${TARGET_PR_REPO_BRANCH}" != "apache:master" ]; then
+ echo "CDLOG Merging not into the master, but into the branch apache/${TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+fi
+
+
+if [ ${PR_TYPE} != "closed" ] || [ ${MERGE_STATUS} != "true" ];
+then
+ echo "CDLOG $(date --utc '+%D %T') PR in $PR_URL for the COMMIT $PR_COMMIT is not in closed state or not merged into the Apache/Beam master repo."
+ exit 0
+fi
+
+
+echo "CDLOG Pull Request $PR_URL has been successfully merged into Apache Beam GitHub repository. Continuing the process."
+
+echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+
+apt update > /dev/null 2>&1
+
+
+apt install -y git curl > /dev/null 2>&1
+
+
+apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+
+echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+
+/etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
Review Comment:
`apt-key` is deprecated. Please do not use `apt-key`.
Please follow "If your distribution (Debian 11+ or Ubuntu 21.10+) doesn't support apt-key, run the following command:" section at https://cloud.google.com/sdk/docs/install
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1104591590
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,165 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Added set -x to show output into cloud build console to the bucket
+set -x
+
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+#Install python java8 and dependencies
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+# Env configuration commands
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null
+apt install python3.8-venv > /dev/null
+pip install -r playground/infrastructure/requirements.txt > /dev/null
+
+# Install jdk and gradle
+apt-get install openjdk-8-jdk -y > /dev/null
+curl -L https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip -o gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+unzip gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+export PATH=$PATH:gradle-${GRADLE_VERSION}/bin > /dev/null
+
+# Install go
+curl -OL https://golang.org/dl/go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+tar -C /usr/local -xvf go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+export PATH=$PATH:/usr/local/go/bin > /dev/null
+
+# Install Docker
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - > /dev/null
+add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+apt update > /dev/null && apt install -y docker-ce > /dev/null
+
+# Assigning required values for CI_CD.py script
+export \
+ORIGIN=PG_EXAMPLES \
+STEP=CI \
+SUBDIRS="./learning/katas ./examples ./sdks" \
+GOOGLE_CLOUD_PROJECT=${PROJECT_ID} \
+BEAM_ROOT_DIR="../.." \
+SDK_CONFIG="../sdks.yaml" \
+BEAM_EXAMPLE_CATEGORIES="../categories.yaml" \
+BEAM_CONCURRENCY=4 \
+BEAM_VERSION=2.43.0 \
+sdks=("java" "python" "go") \
+allowlist=("playground/infrastructure" "playground/backend")
+
+# Check whether commit is tagged
+tag_name=$(git tag --points-at $commit_sha)
+
+# Get diff
+if [ -z $base_ref ] || [ $base_ref == "master" ]
+then
+ base_ref=origin/master
+fi
+diff=$(git diff --name-only $base_ref $commit_sha | tr '\n' ' ')
+
+# Check if there are Examples as
+for sdk in "${sdks[@]}"
+do
+ cd playground/infrastructure
+ python3 checker.py \
+ --verbose \
+ --sdk SDK_"${sdk^^}" \
+ --allowlist "${allowlist[@]}" \
+ --paths ${diff}
+ checker_status=$?
+ cd -
+ if [ $checker_status -eq 0 ]
+ then
+ echo "Checker has found changed examples for ${sdk^^}" >> ${1}
+ example_has_changed=True
+ elif [ $checker_status -eq 11 ]
+ then
+ echo "Checker has not found changed examples for ${sdk^^}" >> ${1}
+ example_has_changed=False
+ else
+ echo "Error: Checker is broken" >> ${1}
+ exit 1
+ fi
+
+# Run main logic if examples have been changed
+ if [[ $example_has_changed == "True" ]]
+ then
+ if [ -z "${tag_name}" ] && [ "${commit_sha}" ]
+ then
+ DOCKERTAG=${commit_sha}
+ elif [ "${tag_name}" ] && [ "${commit_sha}" ]
+ then
+ DOCKERTAG=${tag_name}
+ elif [ "${tag_name}" ] && [ -z "${commit_sha}" ]
+ then
+ DOCKERTAG=${tag_name}
+ elif [ -z "${tag_name}" ] && [ -z "${commit_sha}" ]
+ then
+ echo "Error: DOCKERTAG is empty"
+ exit 1
+ fi
+
+ if [ "$sdk" == "python" ]
+ then
+ # builds apache/beam_python3.7_sdk:$DOCKERTAG image
+ ./gradlew -i :sdks:python:container:py37:docker -Pdocker-tag=${DOCKERTAG}
+ # and set SDK_TAG to DOCKERTAG so that the next step would find it
+ SDK_TAG=${DOCKERTAG}
+ else
+ unset SDK_TAG
+ fi
+
+ opts=" -Pdocker-tag=${DOCKERTAG}"
+ if [ -n "$SDK_TAG" ]
+ then
+ opts="${opts} -Psdk-tag=${SDK_TAG}"
+ fi
+
+ if [ "$sdk" == "java" ]
+ then
+ # Java uses a fixed BEAM_VERSION
+ opts="$opts -Pbase-image=apache/beam_java8_sdk:2.43.0"
+ fi
+
+ ./gradlew -i playground:backend:containers:"${sdk}":docker ${opts}
+
+ IMAGE_TAG=apache/beam_playground-backend-${sdk}:${DOCKERTAG}
+
+
+ docker run -d -p 8080:8080 --network=cloudbuild -e PROTOCOL_TYPE=TCP --name container-${sdk} $IMAGE_TAG
+ sleep 10
+ export SERVER_ADDRESS=container-${sdk}:8080
+ cd playground/infrastructure
+ python3 ci_cd.py \
+ --step ${STEP} \
+ --sdk SDK_"${sdk^^}" \
+ --origin ${ORIGIN} \
+ --subdirs ${SUBDIRS} >> ${1}
Review Comment:
Don't use parameters directly. Create an environment variable with default value. Use correct redirection operator ('>> file 2>&1' instead of '>>' to redirect stdout and stderr streams)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #25117: Playground ci webhook
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25117:
URL: https://github.com/apache/beam/pull/25117#issuecomment-1435239300
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ruslan-ikhsan commented on a diff in pull request #25117: Playground ci webhook
Posted by "ruslan-ikhsan (via GitHub)" <gi...@apache.org>.
ruslan-ikhsan commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1155908566
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
Review Comment:
Update: this is default substitution, from cloud build, I don't see reason why we should check PROJECT_ID for emptiness
@rshamunov
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158541887
##########
playground/infrastructure/cloudbuild/cloudbuild_helm_update.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
Review Comment:
We removed terraform tasks that needs locally installed terraform. Try to remove terraform installation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158628828
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"learning/katas sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CD
+
+LogOutput "Installing python and dependencies."
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install -y --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null 2>&1
+apt install -y python3.8-venv > /dev/null 2>&1
+pip install -r /workspace/beam/playground/infrastructure/requirements.txt > /dev/null 2>&1
+
+LogOutput "Python and dependencies have been successfully installed."
+
+LogOutput "Checking what files were changed in the PR."
+
+git fetch --all > /dev/null 2>&1
+
+diff_log=$(git diff --name-only $DIFF_BASE...forked/$SOURCE_BRANCH)
+diff=($(echo "$diff_log" | tr '\n' ' '))
+LogOutput "List of changed files in PR $SOURCE_BRANCH merging into $DIFF_BASE are:
+$diff_log"
+
+LogOutput "Looking for changes that require CD validation for [$SDKS] SDKs"
+allowlist_array=($ALLOWLIST)
+for sdk in $SDKS
+do
+ eval "example_for_${sdk}_changed"='False'
+ LogOutput "------------------Starting checker.py for SDK_${sdk^^}------------------"
+ cd $BEAM_ROOT_DIR/playground/infrastructure
+ python3 checker.py \
+ --verbose \
+ --sdk SDK_"${sdk^^}" \
+ --allowlist "" \
+ --paths "${diff[@]}"
+ checker_status=$?
+ if [ $checker_status -eq 0 ]; then
+ LogOutput "Checker found changed examples for SDK_${sdk^^}"
+ eval "example_for_${sdk}_changed"='True'
+ elif [ $checker_status -eq 11 ]; then
+ LogOutput "Checker did not find any changed examples for SDK_${sdk^^}"
+ eval "example_for_${sdk}_changed"='False'
+ continue
+ else
+ LogOutput "Error: Checker is broken. Exiting the script."
+ exit 1
+ fi
+
+ result=$(eval echo '$'"example_for_${sdk}_changed")
+
+ if [[ $result == True ]]; then
+ LogOutput "Running ci_cd.py for SDK $sdk"
+
+ export SERVER_ADDRESS=https://${sdk}.${DNS_NAME}
+ python3 ci_cd.py \
Review Comment:
If you are installing custom version of python (3.8) please invoke the desired version directly. As writter here, it's not clear what version of python executable will actually be executed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158602383
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,161 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"learning/katas sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=${STEP}
+ SUBDIRS=$SUBDIRS
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ SDK_CONFIG=$SDK_CONFIG
+ BEAM_EXAMPLE_CATEGORIES=$BEAM_EXAMPLE_CATEGORIES
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ DNS_NAME=$DNS_NAME
+ BEAM_USE_WEBGRPC=$BEAM_USE_WEBGRPC
+ NAMESPACE=$NAMESPACE
+ SOURCE_BRANCH=$SOURCE_BRANCH"
+
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CD
+
+LogOutput "Installing python and dependencies."
+# Install Python 3 and dependencies
+set -e # Exit immediately if any command fails
+apt update > /dev/null 2>&1
+export DEBIAN_FRONTEND=noninteractive
+apt install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null 2>&1
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null 2>&1 && apt update > /dev/null 2>&1
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null 2>&1
+apt install -y --reinstall python3.8-distutils > /dev/null 2>&1
+pip install --upgrade google-api-python-client > /dev/null 2>&1
+python3.8 -m pip install pip --upgrade > /dev/null 2>&1
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null 2>&1
Review Comment:
```suggestion
apt install python-is-python3 > /dev/null 2>&1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158595849
##########
playground/infrastructure/cloudbuild/cloudbuild_helm_update.yaml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk > /dev/null
+
+ git clone --branch merged_pg_gke_deployment
+ https://github.com/akvelon/beam.git
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ cd beam
+
+ mkdir playground/terraform/environment/$_ENVIRONMENT_NAME
+
+ printf '%s\n' 'project_id = "$PROJECT_ID"' 'network_name =
Review Comment:
Please use
```sh
cat > file <<EOF
EOF
```
pattern for pasting large snippets of text into files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158547976
##########
playground/infrastructure/cloudbuild/cloudbuild_helm_update.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk > /dev/null
+
+ git clone --branch merged_pg_gke_deployment
+ https://github.com/akvelon/beam.git
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ cd beam
+
+ mkdir playground/terraform/environment/myenv
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/terraform.tfvars
+ playground/terraform/environment/myenv/terraform.tfvars
+
+ gcloud compute scp --zone=us-central1-a
+ instance-1:/home/ruslan_ikhsanov/beam/playground/terraform/environment/myenv/state.tfbackend
+ playground/terraform/environment/myenv/state.tfbackend
+
+ cat playground/terraform/environment/myenv/terraform.tfvars
+
+ cat playground/terraform/environment/myenv/state.tfbackend
+
+ gcloud auth configure-docker us-east1-docker.pkg.dev
+
+ gcloud container clusters get-credentials --region us-east1-b
+ playground-gke --project cloudbuildtest-382210
+
+ ./gradlew playground:terraform:gkebackend -Pproject_environment=myenv
+ -Pdocker-tag=$_TAG_NAME -Pdns-name=$_DNS_NAME -Psdk-tag=$_SDK_TAG
Review Comment:
This is not Bash script. The yaml syntax is corrected by CloudBuild UI in GCP console. It automatically breaks commands to lines. Commands separated by empty lines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1175150714
##########
playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/triggers.tf:
##########
@@ -63,20 +71,26 @@ resource "google_cloudbuild_trigger" "playground_to_gke" {
description = "Creates cloud build manual trigger to deploy Playground to GKE"
source_to_build {
- uri = "https://github.com/beamplayground/deploy-workaround"
+ uri = var.trigger_source_repo
ref = "refs/heads/master"
repo_type = "GITHUB"
}
-
- git_file_source {
- path = ""
- uri = "https://github.com/beamplayground/deploy-workaround"
- revision = "refs/heads/master"
- repo_type = "GITHUB"
+ build {
+ timeout = "7200s"
+ options {
+ logging = "CLOUD_LOGGING_ONLY"
Review Comment:
options:
machineType: E2_HIGHCPU_32
logging: GCS_ONLY
We want to use better machines than defaut one. We want to log to a GCS private bucket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1175153042
##########
playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/variables.tf:
##########
@@ -162,4 +162,19 @@ variable "data_for_github_pat_secret" {
variable "private_logs_bucket" {
description = "The GCS bucket name to store CI logs privately"
+}
+
+#What i understand this is mandatoy but not actually used. Should we document it
+variable "trigger_source_repo" {
Review Comment:
Add default value:
"https://github.com/beamplayground/deploy-workaround"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] volatilemolotov commented on a diff in pull request #25117: Playground ci webhook
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1175241583
##########
playground/terraform/infrastructure/cloudbuild-manual-setup/02.builders/terraform.tf:
##########
@@ -22,7 +22,7 @@ terraform {
required_providers {
google = {
source = "hashicorp/google"
- version = "~> 4.40.0"
+ version = "~> 4.62.0"
Review Comment:
This is because i needed latest to support `script` field for triggers. Usually I don't recommend to go with latest unless necessary. We should have separate discussion and task for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1104591921
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Install Python 3.8 and dependencies
+apt-get update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+apt-get install -y software-properties-common > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3-pip > /dev/null
+
+cd playground/infrastructure
+pip install -r requirements.txt > /dev/null
+
+export \
+ORIGIN=PG_EXAMPLES \
+STEP=CD \
+SUBDIRS="./learning/katas ./examples ./sdks" \
+GOOGLE_CLOUD_PROJECT=${PROJECT_ID} \
+BEAM_ROOT_DIR="../../" \
+SDK_CONFIG="../sdks.yaml" \
+BEAM_EXAMPLE_CATEGORIES="../categories.yaml" \
+BEAM_USE_WEBGRPC=yes \
+BEAM_CONCURRENCY=2 \
+sdks=("java" "python" "go")
+
+# Run CD script to deploy Examples to Playground for Go, Java, Python SDK
+for sdk in "${sdks[@]}"
+do
+ export SERVER_ADDRESS=https://${sdk}.${dns_name}
+ python3 ci_cd.py \
+ --datastore-project ${PROJECT_ID} \
+ --step ${STEP} \
+ --sdk SDK_"${sdk^^}" \
+ --origin ${ORIGIN} \
+ --subdirs ${SUBDIRS} >> ${1}
Review Comment:
Don't use parameters directly. Create an environment variable with default value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1104653870
##########
playground/infrastructure/cloudbuild/cloudbuild_pg_to_gke.yaml:
##########
@@ -1,48 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
Why did you delete those files? They are deploying PG, not related to CI/CD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1139906312
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,154 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+if [[ -z "${PROJECT_ID}" ]]; then
+ echo "PROJECT_ID is empty or not set. Exiting"
+ exit 1
+fi
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend sdks/ examples/"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
+ ORIGIN=$ORIGIN
+ STEP=$STEP
Review Comment:
You didn't set a value for the variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ruslan-ikhsan commented on a diff in pull request #25117: Playground ci webhook
Posted by "ruslan-ikhsan (via GitHub)" <gi...@apache.org>.
ruslan-ikhsan commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138389267
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
Review Comment:
Check if variable is not empty condition added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1104591590
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,165 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Added set -x to show output into cloud build console to the bucket
+set -x
+
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+#Install python java8 and dependencies
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+# Env configuration commands
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null
+apt install python3.8-venv > /dev/null
+pip install -r playground/infrastructure/requirements.txt > /dev/null
+
+# Install jdk and gradle
+apt-get install openjdk-8-jdk -y > /dev/null
+curl -L https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip -o gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+unzip gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+export PATH=$PATH:gradle-${GRADLE_VERSION}/bin > /dev/null
+
+# Install go
+curl -OL https://golang.org/dl/go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+tar -C /usr/local -xvf go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+export PATH=$PATH:/usr/local/go/bin > /dev/null
+
+# Install Docker
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - > /dev/null
+add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+apt update > /dev/null && apt install -y docker-ce > /dev/null
+
+# Assigning required values for CI_CD.py script
+export \
+ORIGIN=PG_EXAMPLES \
+STEP=CI \
+SUBDIRS="./learning/katas ./examples ./sdks" \
+GOOGLE_CLOUD_PROJECT=${PROJECT_ID} \
+BEAM_ROOT_DIR="../.." \
+SDK_CONFIG="../sdks.yaml" \
+BEAM_EXAMPLE_CATEGORIES="../categories.yaml" \
+BEAM_CONCURRENCY=4 \
+BEAM_VERSION=2.43.0 \
+sdks=("java" "python" "go") \
+allowlist=("playground/infrastructure" "playground/backend")
+
+# Check whether commit is tagged
+tag_name=$(git tag --points-at $commit_sha)
+
+# Get diff
+if [ -z $base_ref ] || [ $base_ref == "master" ]
+then
+ base_ref=origin/master
+fi
+diff=$(git diff --name-only $base_ref $commit_sha | tr '\n' ' ')
+
+# Check if there are Examples as
+for sdk in "${sdks[@]}"
+do
+ cd playground/infrastructure
+ python3 checker.py \
+ --verbose \
+ --sdk SDK_"${sdk^^}" \
+ --allowlist "${allowlist[@]}" \
+ --paths ${diff}
+ checker_status=$?
+ cd -
+ if [ $checker_status -eq 0 ]
+ then
+ echo "Checker has found changed examples for ${sdk^^}" >> ${1}
+ example_has_changed=True
+ elif [ $checker_status -eq 11 ]
+ then
+ echo "Checker has not found changed examples for ${sdk^^}" >> ${1}
+ example_has_changed=False
+ else
+ echo "Error: Checker is broken" >> ${1}
+ exit 1
+ fi
+
+# Run main logic if examples have been changed
+ if [[ $example_has_changed == "True" ]]
+ then
+ if [ -z "${tag_name}" ] && [ "${commit_sha}" ]
+ then
+ DOCKERTAG=${commit_sha}
+ elif [ "${tag_name}" ] && [ "${commit_sha}" ]
+ then
+ DOCKERTAG=${tag_name}
+ elif [ "${tag_name}" ] && [ -z "${commit_sha}" ]
+ then
+ DOCKERTAG=${tag_name}
+ elif [ -z "${tag_name}" ] && [ -z "${commit_sha}" ]
+ then
+ echo "Error: DOCKERTAG is empty"
+ exit 1
+ fi
+
+ if [ "$sdk" == "python" ]
+ then
+ # builds apache/beam_python3.7_sdk:$DOCKERTAG image
+ ./gradlew -i :sdks:python:container:py37:docker -Pdocker-tag=${DOCKERTAG}
+ # and set SDK_TAG to DOCKERTAG so that the next step would find it
+ SDK_TAG=${DOCKERTAG}
+ else
+ unset SDK_TAG
+ fi
+
+ opts=" -Pdocker-tag=${DOCKERTAG}"
+ if [ -n "$SDK_TAG" ]
+ then
+ opts="${opts} -Psdk-tag=${SDK_TAG}"
+ fi
+
+ if [ "$sdk" == "java" ]
+ then
+ # Java uses a fixed BEAM_VERSION
+ opts="$opts -Pbase-image=apache/beam_java8_sdk:2.43.0"
+ fi
+
+ ./gradlew -i playground:backend:containers:"${sdk}":docker ${opts}
+
+ IMAGE_TAG=apache/beam_playground-backend-${sdk}:${DOCKERTAG}
+
+
+ docker run -d -p 8080:8080 --network=cloudbuild -e PROTOCOL_TYPE=TCP --name container-${sdk} $IMAGE_TAG
+ sleep 10
+ export SERVER_ADDRESS=container-${sdk}:8080
+ cd playground/infrastructure
+ python3 ci_cd.py \
+ --step ${STEP} \
+ --sdk SDK_"${sdk^^}" \
+ --origin ${ORIGIN} \
+ --subdirs ${SUBDIRS} >> ${1}
Review Comment:
Don't use parameters directly. Create an environment variable with default value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138189168
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ 1. PR URL: $_PR_URL;
+ 2. PR Source Repo: $_FORK_REPO;
+ 3. PR Branch: $_PR_BRANCH;
+ 4. PR Commit: $_PR_COMMIT;
+ 5. WebHook Action: $_PR_TYPE;
+ 6. Merged to: $_TARGET_PR_REPO_BRANCH;
+ 7. Playground DNS: $_DNS_NAME; "
+ if [[ "${_PR_BRANCH}" != "playground-ci-webhook" ]]; then
+ exit 0
+ fi
+ if [[ ${_TARGET_PR_REPO_BRANCH} == "apache:master" ]]; then
+ echo "CDLOG $(date --utc '+%D %T').
+
+ Pull Request $_PR_URL has been successfully merged into Apache/Beam Master repository (https://github.com/apache/beam).
+
+ Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git > /dev/null 2>&1
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/akvelon/beam/playground-ci-webhook/playground/infrastructure/cloudbuild/examples_cd.sh
Review Comment:
replace to apache master path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ruslan-ikhsan commented on a diff in pull request #25117: Playground ci webhook
Posted by "ruslan-ikhsan (via GitHub)" <gi...@apache.org>.
ruslan-ikhsan commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138361740
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ 1. PR URL: $_PR_URL;
+ 2. PR Source Repo: $_FORK_REPO;
+ 3. PR Branch: $_PR_BRANCH;
+ 4. PR Commit: $_PR_COMMIT;
+ 5. WebHook Action: $_PR_TYPE;
+ 6. Merged to: $_TARGET_PR_REPO_BRANCH;
+ 7. Playground DNS: $_DNS_NAME; "
+ if [[ "${_PR_BRANCH}" != "playground-ci-webhook" ]]; then
+ exit 0
+ fi
+ if [[ ${_TARGET_PR_REPO_BRANCH} == "apache:master" ]]; then
+ echo "CDLOG $(date --utc '+%D %T').
+
+ Pull Request $_PR_URL has been successfully merged into Apache/Beam Master repository (https://github.com/apache/beam).
+
+ Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git > /dev/null 2>&1
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/akvelon/beam/playground-ci-webhook/playground/infrastructure/cloudbuild/examples_cd.sh
+
+ chmod +x /tmp/examples_cd.sh
+
+ env -i bash -c "/tmp/examples_cd.sh PROJECT_ID='"'${PROJECT_ID}'"' DNS_NAME='"'${_DNS_NAME}'"' COMMIT='"'${_PR_COMMIT}'"' "
Review Comment:
@rshamunov added the parameter and all required logic elsewhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1139935882
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,154 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+if [[ -z "${PROJECT_ID}" ]]; then
+ echo "PROJECT_ID is empty or not set. Exiting"
+ exit 1
+fi
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+if [[ -z "${DNS_NAME}" ]]; then
+ echo "DNS_NAME is empty or not set. Exiting"
+ exit 1
+fi
+export NAMESPACE=${NAMESPACE}
+if [[ -z "${NAMESPACE}" ]]; then
+ echo "Datastore NAMESPACE is empty or not set. Exiting"
+ exit 1
+fi
+export SOURCE_BRANCH=${SOURCE_BRANCH}
+if [[ -z "${SOURCE_BRANCH}" ]]; then
+ echo "PR Source Branch is empty or not set. Exiting"
+ exit 1
+fi
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend sdks/ examples/"}
Review Comment:
Remove ALLOWLIST from CD script
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ruslan-ikhsan commented on a diff in pull request #25117: Playground ci webhook
Posted by "ruslan-ikhsan (via GitHub)" <gi...@apache.org>.
ruslan-ikhsan commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1138391201
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export SDK_CONFIG=${SDK_CONFIG-"$BEAM_ROOT_DIR/playground/sdks.yaml"}
+export BEAM_EXAMPLE_CATEGORIES=${BEAM_EXAMPLE_CATEGORIES-"$BEAM_ROOT_DIR/playground/categories.yaml"}
+export BEAM_USE_WEBGRPC=${BEAM_USE_WEBGRPC-"yes"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-2}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DNS_NAME=${DNS_NAME}
+
+# This function logs the given message to a file and outputs it to the console.
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CDLOG keyword to simplify search over the global log
+ echo "CDLOG $(date --utc '+%D %T') $1"
+}
+
+LogOutput "Input variables:
Review Comment:
Removed the row numbers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ruslan-ikhsan commented on a diff in pull request #25117: Playground ci webhook
Posted by "ruslan-ikhsan (via GitHub)" <gi...@apache.org>.
ruslan-ikhsan commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1155912267
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL;
+ PR Source Repo: $_FORK_REPO;
+ PR Branch: $_PR_BRANCH;
+ PR Commit: $_PR_COMMIT;
+ WebHook Action: $_PR_TYPE;
+ Merged to: $_TARGET_PR_REPO_BRANCH;
+ Playground DNS: $_DNS_NAME; "
+ if [[ ${_TARGET_PR_REPO_BRANCH} != "apache:master" ]]; then
+ echo "CDLOG Merge to branch apache/${_TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == "closed" ]] && [[ ${_MERGE_STATUS} == "true" ]]; then
+
+ echo "CDLOG Pull Request $_PR_URL has been successfully closed and merged into Apache/Beam GitHub repository. Continuing the process."
+
+ echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git > /dev/null 2>&1
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/akvelon/beam/master/playground/infrastructure/cloudbuild/examples_cd.sh
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158714229
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,244 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Script validates examples against runners (Java, Python, Go) built as local containers from a current branch
+#
+# Command line arguments:
+# LOG_PATH - full path lo a log file. Output is also logged to stdout (Default: /dev/null)
+# BEAM_ROOT_DIR - Path (full) to cloned repo root. Used by ci_cd.py (Default: /workspace/beam)
+# PROJECT_ID - GCP Project ID. Used by ci_cd.py (Default: test)
+# BEAM_CONCURRENCY - Number of examples to run in parallel. Used by ci_cd.py (Default: 4)
+# BEAM_VERSION - version of BEAM SDK to build containers
+# SUBDIRS - array of paths (relative to beam repo root) to search changed examples (Default: ./learning/katas ./examples ./sdks)
+# COMMIT - Git commit hash to build containers from (Default: HEAD)
+# DIFF_BASE - Git branch to compare with $COMMIT to look for changed examples (Default: origin/master)
+# SDKS - Array of SDKS to validate (Default: java python go)
+# ORIGIN - examples origin (Default: PG_EXAMPLES)
+# ALLOWLIST - List of paths (relative to the repo root) not in examples (SUBDIRS) that cause reb
+
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+
+ export "$KEY"="$VALUE"
+done
+
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export BEAM_VERSION=${BEAM_VERSION-"2.44.0"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-"4"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend"}
+
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CILOG keyword to simplify search over the global log
+ echo "CILOG $(date --utc '+%D %T') $1"
+}
+
+
+LogOutput "Input variables:
+ LOG_PATH=$LOG_PATH
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ BEAM_VERSION=$BEAM_VERSION
+ SUBDIRS=$SUBDIRS
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ ORIGIN=$ORIGIN
+ ALLOWLIST=$ALLOWLIST"
+
+# Assigning constant values
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CI
+export SDK_CONFIG="$BEAM_ROOT_DIR/playground/sdks.yaml"
+export BEAM_EXAMPLE_CATEGORIES="$BEAM_ROOT_DIR/playground/categories.yaml"
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+LogOutput "Installing python java8 and dependencies"
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+LogOutput "Installing Python environment"
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install -y --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null
+apt install -y python3.8-venv > /dev/null
+pip install -r playground/infrastructure/requirements.txt > /dev/null
Review Comment:
Let's keep it as is for now. It's improvement for future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158715258
##########
playground/infrastructure/cloudbuild/playground_cd_stable.yaml:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+ # contributor license agreements. See the NOTICE file distributed with
+ # this work for additional information regarding copyright ownership.
+ # The ASF licenses this file to You under the Apache License, Version 2.0
+ # (the "License"); you may not use this file except in compliance with
+ # the License. You may obtain a copy of the License at
+ #
+ # http://www.apache.org/licenses/LICENSE-2.0
+ #
+ # Unless required by applicable law or agreed to in writing, software
+ # distributed under the License is distributed on an "AS IS" BASIS,
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - |-
+ echo "Search for CDLOG keyword to find valuable logs entries"
+ echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $_PR_URL;
+ PR Source Repo: $_FORK_REPO;
+ PR Branch: $_PR_BRANCH;
+ PR Commit: $_PR_COMMIT;
+ WebHook Action: $_PR_TYPE;
+ Merged to: $_TARGET_PR_REPO_BRANCH;
+ Playground DNS: $_DNS_NAME; "
+ if [[ ${_TARGET_PR_REPO_BRANCH} != "apache:master" ]]; then
+ echo "CDLOG Merging not into the master, but into the branch apache/${_TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+ fi
+
+ if [[ ${_PR_TYPE} == "closed" ]] && [[ ${_MERGE_STATUS} == "true" ]]; then
+
+ echo "CDLOG Pull Request $_PR_URL has been successfully closed and merged into Apache/Beam GitHub repository. Continuing the process."
+
+ echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+ apt update > /dev/null 2>&1
+
+ apt install -y git curl > /dev/null 2>&1
+
+ apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+ apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+ git clone --branch master https://github.com/apache/beam.git > /dev/null 2>&1
+
+ cd beam
+
+ git remote add forked https://github.com/${_FORK_REPO}.git > /dev/null 2>&1
+
+ curl -O --output-dir /tmp https://raw.githubusercontent.com/apache/beam/master/playground/infrastructure/cloudbuild/examples_cd.sh
Review Comment:
No need to fetch for CD, since it runs from master branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158556631
##########
playground/infrastructure/cloudbuild/examples_cd.sh:
##########
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This loop reads the arguments passed to the script, parses them, and exports them as environment variables.
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+ export "$KEY"="$VALUE"
+done
+
+# This block sets default values for several environment variables to run CD script.
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export STEP=${STEP-"CD"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
Review Comment:
We clean up environment variables when we call the script
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158598513
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_infra_part2.yaml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk kubectl > /dev/null
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ git clone --branch merged_pg_gke_deployment
+ https://github.com/akvelon/beam.git
Review Comment:
I don't like that we are hardcoding the repo and branch name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158614848
##########
playground/infrastructure/cloudbuild/examples_ci.sh:
##########
@@ -0,0 +1,244 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Script validates examples against runners (Java, Python, Go) built as local containers from a current branch
+#
+# Command line arguments:
+# LOG_PATH - full path lo a log file. Output is also logged to stdout (Default: /dev/null)
+# BEAM_ROOT_DIR - Path (full) to cloned repo root. Used by ci_cd.py (Default: /workspace/beam)
+# PROJECT_ID - GCP Project ID. Used by ci_cd.py (Default: test)
+# BEAM_CONCURRENCY - Number of examples to run in parallel. Used by ci_cd.py (Default: 4)
+# BEAM_VERSION - version of BEAM SDK to build containers
+# SUBDIRS - array of paths (relative to beam repo root) to search changed examples (Default: ./learning/katas ./examples ./sdks)
+# COMMIT - Git commit hash to build containers from (Default: HEAD)
+# DIFF_BASE - Git branch to compare with $COMMIT to look for changed examples (Default: origin/master)
+# SDKS - Array of SDKS to validate (Default: java python go)
+# ORIGIN - examples origin (Default: PG_EXAMPLES)
+# ALLOWLIST - List of paths (relative to the repo root) not in examples (SUBDIRS) that cause reb
+
+for ARGUMENT in "$@"
+do
+ KEY=$(echo $ARGUMENT | cut -f1 -d=)
+
+ KEY_LENGTH=${#KEY}
+ VALUE="${ARGUMENT:$KEY_LENGTH+1}"
+
+ export "$KEY"="$VALUE"
+done
+
+export LOG_PATH=${LOG_PATH-"/dev/null"}
+export BEAM_ROOT_DIR=${BEAM_ROOT_DIR-"/workspace/beam"}
+export PROJECT_ID=${PROJECT_ID}
+export BEAM_VERSION=${BEAM_VERSION-"2.44.0"}
+export SUBDIRS=${SUBDIRS-"./learning/katas ./examples ./sdks"}
+export SDKS=${SDKS-"java python go"}
+export COMMIT=${COMMIT-"HEAD"}
+export DIFF_BASE=${DIFF_BASE-"origin/master"}
+export BEAM_CONCURRENCY=${BEAM_CONCURRENCY-"4"}
+export ORIGIN=${ORIGIN-"PG_EXAMPLES"}
+export ALLOWLIST=${ALLOWLIST-"playground/infrastructure playground/backend"}
+
+function LogOutput ()
+{
+ echo "$(date --utc '+%D %T') $1" >> $LOG_PATH
+ # CILOG keyword to simplify search over the global log
+ echo "CILOG $(date --utc '+%D %T') $1"
+}
+
+
+LogOutput "Input variables:
+ LOG_PATH=$LOG_PATH
+ BEAM_ROOT_DIR=$BEAM_ROOT_DIR
+ PROJECT_ID=$PROJECT_ID
+ BEAM_VERSION=$BEAM_VERSION
+ SUBDIRS=$SUBDIRS
+ SDKS=$SDKS
+ COMMIT=$COMMIT
+ BEAM_CONCURRENCY=$BEAM_CONCURRENCY
+ ORIGIN=$ORIGIN
+ ALLOWLIST=$ALLOWLIST"
+
+# Assigning constant values
+# Script starts in a clean environment in Cloud Build. Set minimal required environment variables
+if [ -z "$PATH" ]; then
+ export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin/:/usr/bin:/sbin:/bin"
+fi
+if [ -z "$HOME" ]; then
+ export HOME="/builder/home"
+fi
+
+export STEP=CI
+export SDK_CONFIG="$BEAM_ROOT_DIR/playground/sdks.yaml"
+export BEAM_EXAMPLE_CATEGORIES="$BEAM_ROOT_DIR/playground/categories.yaml"
+export GRADLE_VERSION=7.5.1
+export GO_VERSION=1.18
+
+LogOutput "Installing python java8 and dependencies"
+apt-get update > /dev/null
+apt update > /dev/null
+export DEBIAN_FRONTEND=noninteractive
+
+LogOutput "Installing Python environment"
+apt-get install -y apt-transport-https ca-certificates software-properties-common curl unzip apt-utils > /dev/null
+add-apt-repository -y ppa:deadsnakes/ppa > /dev/null && apt update > /dev/null
+apt install -y python3.8 python3.8-distutils python3-pip > /dev/null
+apt install -y --reinstall python3.8-distutils > /dev/null
+pip install --upgrade google-api-python-client > /dev/null
+python3.8 -m pip install pip --upgrade > /dev/null
+ln -s /usr/bin/python3.8 /usr/bin/python > /dev/null
+apt install -y python3.8-venv > /dev/null
+pip install -r playground/infrastructure/requirements.txt > /dev/null
+
+LogOutput "Installing JDK and Gradle"
+apt-get install openjdk-8-jdk -y > /dev/null
+curl -L https://services.gradle.org/distributions/gradle-${GRADLE_VERSION}-bin.zip -o gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+unzip gradle-${GRADLE_VERSION}-bin.zip > /dev/null
+export PATH=$PATH:gradle-${GRADLE_VERSION}/bin > /dev/null
+
+LogOutput "Installing GO"
+curl -OL https://golang.org/dl/go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+tar -C /usr/local -xvf go$GO_VERSION.linux-amd64.tar.gz > /dev/null
+export PATH=$PATH:/usr/local/go/bin > /dev/null
+# required to build beam/sdks/go container
+export GOMODCACHE="$HOME/go/pkg/mod"
+
+LogOutput "Installing Docker"
+curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - > /dev/null
+add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+apt update > /dev/null && apt install -y docker-ce > /dev/null
+
+tag_name=$(git tag --points-at $COMMIT)
+if [ "$tag_name" ]
+then
+ LogOutput "Commit $COMMIT is tagged with $tag_name"
+ DOCKERTAG=$tag_name
+else
+ LogOutput "Commit $COMMIT is not tagged"
+ DOCKERTAG=$COMMIT
+fi
+LogOutput "Docker tag for containers: $DOCKERTAG"
+
+LogOutput "git fetch -all"
+git fetch --all > /dev/null
+LogOutput "git checkout $COMMIT"
+git checkout $COMMIT
+if [ $? -ne 0 ]; then
+ LogOutput "Can't checkout to $COMMIT. Exiting script"
+ exit 1
+fi
+
+diff_log=$(git diff --name-only $DIFF_BASE...$COMMIT)
+diff=($(echo "$diff_log" | tr '\n' ' '))
+LogOutput "Discovered changes introduced by $COMMIT relative to $DIFF_BASE in files:
+$diff_log"
+
+LogOutput "Looking for changes that require CI validation for [$SDKS] SDKs"
+allowlist_array=($ALLOWLIST)
+for sdk in $SDKS
+do
+ eval "ci_${sdk}_passed"='False'
+ example_has_changed="UNKNOWN"
+ LogOutput "------------------Starting checker.py for SDK_${sdk^^}------------------"
+ cd $BEAM_ROOT_DIR/playground/infrastructure
+ python3 checker.py \
+ --verbose \
+ --sdk SDK_"${sdk^^}" \
+ --allowlist "${allowlist_array[@]}" \
+ --paths "${diff[@]}" >> ${LOG_PATH} 2>&1
+ checker_status=$?
+ cd $BEAM_ROOT_DIR
+ if [ $checker_status -eq 0 ]
+ then
+ LogOutput "Checker found changed examples for SDK_${sdk^^}"
+ example_has_changed=True
+ elif [ $checker_status -eq 11 ]
+ then
+ LogOutput "Checker did not find any changed examples for SDK_${sdk^^}"
+ example_has_changed=False
+ else
+ LogOutput "Error: Checker is broken. Exiting the script."
+ exit 1
+ fi
+
+ #Nothing to check
+ if [[ $example_has_changed != "True" ]]
+ then
+ LogOutput "No changes require validation for SDK_${sdk^^}"
+ continue
+ fi
+
+ # docker_options="-Psdk-tag=${DOCKERTAG}"
+ sdk_tag=$BEAM_VERSION
+
+ # Special cases for Python and Java
+ if [ "$sdk" == "python" ]
+ then
+ # Build fails without docker-pull-licenses=true in Cloud Build
+ LogOutput "Building Python base image container apache/beam_python3.7_sdk:$DOCKERTAG -Pdocker-pull-licenses=true"
+ LogOutput "./gradlew -i :sdks:python:container:py37:docker -Pdocker-tag=$DOCKERTAG -Pdocker-pull-licenses=true"
+ sdk_tag=$DOCKERTAG
+ ./gradlew -i :sdks:python:container:py37:docker -Pdocker-tag=$DOCKERTAG #-Pdocker-pull-licenses=true
+ if [ $? -ne 0 ]
+ then
+ LogOutput "Build failed for apache/beam_python3.7_sdk:$DOCKERTAG"
+ continue
+ fi
+ fi
+
+ LogOutput "Buidling a container for $sdk runner"
+ LogOutput "./gradlew -i playground:backend:containers:"${sdk}":docker -Psdk-tag=$sdk_tag -Pdocker-tag=$DOCKERTAG"
+ ./gradlew -i playground:backend:containers:"${sdk}":docker -Psdk-tag=$sdk_tag -Pdocker-tag=$DOCKERTAG
+ if [ $? -ne 0 ]
+ then
+ LogOutput "Container build failed for $sdk runner"
+ continue
+ fi
+ LogOutput "Starting container for $sdk runner"
+ docker run -d -p 8080:8080 --network=cloudbuild -e PROTOCOL_TYPE=TCP --name container-${sdk} apache/beam_playground-backend-${sdk}:$DOCKERTAG
+ sleep 10
+ export SERVER_ADDRESS=container-${sdk}:8080
+
+ LogOutput "Starting ci_cd.py to validate ${sdk} examples"
+ cd $BEAM_ROOT_DIR/playground/infrastructure
+ python3 ci_cd.py \
+ --step ${STEP} \
+ --sdk SDK_"${sdk^^}" \
+ --origin ${ORIGIN} \
+ --subdirs ${SUBDIRS} >> ${LOG_PATH} 2>&1
+ if [ $? -eq 0 ]
+ then
+ LogOutput "Example validation for $sdk SDK successfully completed"
+ eval "ci_${sdk}_passed"='True'
+ else
+ LogOutput "Example validation for $sdk SDK failed"
+ fi
+ cd $BEAM_ROOT_DIR
+ LogOutput "Stopping container for $sdk runner"
+ docker stop container-${sdk} > /dev/null
+ docker rm container-${sdk} > /dev/null
+done
+
+LogOutput "Script finished"
+for sdk in $SDKS
+do
+ result=$(eval echo '$'"ci_${sdk}_passed")
+ if [ "$result" != "True" ]; then
+ "At least one of the checks has failed"
+ exit 1
+ fi
+done
+exit 0
Review Comment:
```suggestion
exit 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov commented on a diff in pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1158691464
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_infra_part2.yaml:
##########
@@ -0,0 +1,101 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+ - name: ubuntu
+ args:
+ - '-c'
+ - >-
+ apt update > /dev/null
+
+ apt-get install -y wget unzip software-properties-common git curl
+ apt-transport-https ca-certificates gnupg jq > /dev/null
+
+ echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | tee -a
+ /etc/apt/sources.list.d/google-cloud-sdk.list
+
+ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add
+ -
+
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
+
+ add-apt-repository "deb [arch=amd64]
+ https://download.docker.com/linux/ubuntu focal stable" > /dev/null
+
+ apt install -y docker-ce > /dev/null
+
+ wget
+ https://releases.hashicorp.com/terraform/1.4.2/terraform_1.4.2_linux_amd64.zip
+
+ unzip terraform_1.4.2_linux_amd64.zip
+
+ mv terraform /usr/local/bin/terraform
+
+ terraform --version
+
+ apt update -y > /dev/null && apt install -y google-cloud-sdk
+ openjdk-11-jdk kubectl > /dev/null
+
+ curl -fsSLo get_helm.sh
+ https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 >
+ /dev/null
+
+ chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+ apt-get install google-cloud-sdk-gke-gcloud-auth-plugin
+
+ git clone --branch merged_pg_gke_deployment
Review Comment:
We should be able to change repo and branch for deployment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TSultanov commented on a diff in pull request #25117: Playground ci webhook
Posted by "TSultanov (via GitHub)" <gi...@apache.org>.
TSultanov commented on code in PR #25117:
URL: https://github.com/apache/beam/pull/25117#discussion_r1178840291
##########
playground/infrastructure/cloudbuild/cloudbuild_playground_cd_examples.sh:
##########
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#!/usr/bin/env bash
+echo "Search for CDLOG keyword to find valuable logs entries"
+echo "CDLOG $(date --utc '+%D %T') Trigger run inputs:
+ PR URL: $PR_URL
+ Merged to: $TARGET_PR_REPO_BRANCH
+ WebHook Action: $PR_TYPE
+ Merge Status: $MERGE_STATUS
+ Merge commit: $MERGE_COMMIT
+ Playground DNS: $DNS_NAME
+ Datastore namespace: $DATASTORE_NAMESPACE
+ Exampes Origin: $ORIGIN
+ Examples Subdirs: $SUBDIRS
+ SDKs: $SDKS
+ CD Script Concurrency: $BEAM_CONCURRENCY"
+
+if [ -z "$DNS_NAME" ] || [ -z "$DATASTORE_NAMESPACE" ]; then
+ echo "DNS_NAME and DATASTORE_NAMESPACE substitutions must be set in the triger"
+ exit 1
+fi
+
+
+if [ "${TARGET_PR_REPO_BRANCH}" != "apache:master" ]; then
+ echo "CDLOG Merging not into the master, but into the branch apache/${TARGET_PR_REPO_BRANCH}. Exiting"
+ exit 0
+fi
+
+
+if [ ${PR_TYPE} != "closed" ] || [ ${MERGE_STATUS} != "true" ];
+then
+ echo "CDLOG $(date --utc '+%D %T') PR in $PR_URL for the COMMIT $PR_COMMIT is not in closed state or not merged into the Apache/Beam master repo."
+ exit 0
+fi
+
+
+echo "CDLOG Pull Request $PR_URL has been successfully merged into Apache Beam GitHub repository. Continuing the process."
+
+echo "CDLOG Continous Deployment of Playground Examples (CD) in the progress."
+
+
+apt update > /dev/null 2>&1
+
+
+apt install -y git curl > /dev/null 2>&1
+
+
+apt-get install -y apt-transport-https ca-certificates gnupg > /dev/null 2>&1
+
+
+echo "deb https://packages.cloud.google.com/apt cloud-sdk main" > /dev/null 2>&1 | tee -a
+
+/etc/apt/sources.list.d/google-cloud-sdk.list > /dev/null 2>&1
+
+curl https://packages.cloud.google.com/apt/doc/apt-key.gpg > /dev/null 2>&1 | apt-key add - > /dev/null 2>&1
+
+
+apt-get update && apt-get install -y google-cloud-sdk > /dev/null 2>&1
+
+echo "CDLOG $(date --utc '+%D %T') Starting deployment script"
+
+git clone --branch master https://github.com/apache/beam.git
+
+chmod +x ${CD_SCRIPT_PATH}
+
+bash -c "${CD_SCRIPT_PATH}"
Review Comment:
What's the rationale behind using `-c`?
`./${CD_SCRIPT_PATH}` should be enough given that the executable bit was set on the file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rshamunov closed pull request #25117: Playground ci webhook
Posted by "rshamunov (via GitHub)" <gi...@apache.org>.
rshamunov closed pull request #25117: Playground ci webhook
URL: https://github.com/apache/beam/pull/25117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org