You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/01/06 21:56:19 UTC

[GitHub] [beam] youngoli opened a new pull request #13678: [BEAM-11415] Add Dataflow Runner to new Go ValidatesRunner framework.

youngoli opened a new pull request #13678:
URL: https://github.com/apache/beam/pull/13678


   Straightforward, this adds Dataflow support to the script, as well as a corresponding Gradle command for Dataflow.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam
 .apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.a
 pache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam
 .apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website | Whitespace | Typescript
   --- | --- | --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/be
 am_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] lostluck commented on a change in pull request #13678: [BEAM-11415] Add Dataflow Runner to new Go ValidatesRunner framework.

Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #13678:
URL: https://github.com/apache/beam/pull/13678#discussion_r552986389



##########
File path: sdks/go/test/run_validatesrunner_tests.sh
##########
@@ -16,39 +16,68 @@
 #    limitations under the License.
 
 # This script executes ValidatesRunner tests including launching any additional
-# services needed, such as job services or expansion services. The following
-# runners are supported, and selected via a flag:
+# services needed, such as job services or expansion services. This script
+# should be executed from the root of the Beam repository.
 #
+# The following runners are supported, and selected via a flag:
 # --runner {portable|direct|flink} (default: portable)
 #  Select which runner to execute tests on. This flag also determines which
 #  services to start up and which tests may be skipped.
 #    direct   - Go SDK Direct Runner
 #    portable - (default) Python Portable Runner (aka. Reference Runner or FnAPI Runner)
 #    flink    - Java Flink Runner (local mode)
 #    spark    - Java Spark Runner (local mode)
+#    dataflow - Dataflow Runner
 #
-# --flink_job_server_jar -> Filepath to jar, used if runner is Flink.
-# --spark_job_server_jar -> Filepath to jar, used if runner is Spark.
-# --endpoint -> Replaces jar filepath with existing job server endpoint.
+# General flags:
+#    --timeout -> Timeout for the go test command, on a per-package level.
+#    --endpoint -> An endpoint for an existing job server outside the script.
+#        If present, job server jar flags are ignored.
+#    --expansion_service_jar -> Filepath to jar for expansion service, for
+#        runners that support cross-language.
+#    --expansion_addr -> An endpoint for an existing expansion service outside
+#        the script. If present, expansion_service_jar is ignored.
 #
-# --expansion_service_jar -> Filepath to jar for expansion service.
-# --expansion_addr -> Replaces jar filepath with existing expansion service endpoint.
-#
-# Execute from the root of the repository. This script requires that necessary
-# services can be built from the repository.
+# Runner-specific flags:
+#  Flink
+#    --flink_job_server_jar -> Filepath to jar, used if runner is Flink.
+#  Spark
+#    --spark_job_server_jar -> Filepath to jar, used if runner is Spark.
+#  Dataflow
+#    --dataflow_project -> GCP project to run Dataflow jobs on.
+#    --project -> Same project as dataflow-project, but in URL format, for
+#        example in the format "us.gcr.io/<project>".
+#    --region -> GCP region to run Dataflow jobs on.
+#    --gcs_location -> GCS URL for storing temporary files for Dataflow jobs.
+#    --dataflow_worker_jar -> The Dataflow worker jar to use when running jobs.
+#        If not specified, the script attempts to retrieve a previously built
+#        jar from the appropriate gradle module, which may not succeed.
 
 set -e
 set -v
 
+# Default runner.
 RUNNER=portable
 
+# Default timeout. This timeout is applied per-package, as tests in different
+# packages are executed in parallel.
+TIMEOUT=1h
+
+# Where to store integration test outputs.
+GCS_LOCATION=gs://clouddfe-danoliveira/temp-storage-go-end-to-end-tests

Review comment:
       Definitely will need to change this to something more jenkins safe. And the defaults below.

##########
File path: sdks/go/test/run_validatesrunner_tests.sh
##########
@@ -16,39 +16,68 @@
 #    limitations under the License.
 
 # This script executes ValidatesRunner tests including launching any additional
-# services needed, such as job services or expansion services. The following
-# runners are supported, and selected via a flag:
+# services needed, such as job services or expansion services. This script
+# should be executed from the root of the Beam repository.
 #
+# The following runners are supported, and selected via a flag:
 # --runner {portable|direct|flink} (default: portable)
 #  Select which runner to execute tests on. This flag also determines which
 #  services to start up and which tests may be skipped.
 #    direct   - Go SDK Direct Runner
 #    portable - (default) Python Portable Runner (aka. Reference Runner or FnAPI Runner)
 #    flink    - Java Flink Runner (local mode)
 #    spark    - Java Spark Runner (local mode)
+#    dataflow - Dataflow Runner
 #
-# --flink_job_server_jar -> Filepath to jar, used if runner is Flink.
-# --spark_job_server_jar -> Filepath to jar, used if runner is Spark.
-# --endpoint -> Replaces jar filepath with existing job server endpoint.
+# General flags:
+#    --timeout -> Timeout for the go test command, on a per-package level.
+#    --endpoint -> An endpoint for an existing job server outside the script.
+#        If present, job server jar flags are ignored.
+#    --expansion_service_jar -> Filepath to jar for expansion service, for
+#        runners that support cross-language.
+#    --expansion_addr -> An endpoint for an existing expansion service outside
+#        the script. If present, expansion_service_jar is ignored.
 #
-# --expansion_service_jar -> Filepath to jar for expansion service.
-# --expansion_addr -> Replaces jar filepath with existing expansion service endpoint.
-#
-# Execute from the root of the repository. This script requires that necessary
-# services can be built from the repository.
+# Runner-specific flags:
+#  Flink
+#    --flink_job_server_jar -> Filepath to jar, used if runner is Flink.
+#  Spark
+#    --spark_job_server_jar -> Filepath to jar, used if runner is Spark.
+#  Dataflow
+#    --dataflow_project -> GCP project to run Dataflow jobs on.
+#    --project -> Same project as dataflow-project, but in URL format, for

Review comment:
       ```suggestion
   #    --project -> Same project as --dataflow_project, but in URL format, for
   ```

##########
File path: sdks/go/test/run_validatesrunner_tests.sh
##########
@@ -151,11 +222,70 @@ if [[ "$RUNNER" == "flink" || "$RUNNER" == "spark" || "$RUNNER" == "portable" ]]
     java -jar $EXPANSION_SERVICE_JAR $EXPANSION_PORT &
     EXPANSION_PID=$!
   fi
+fi
+
+if [[ "$RUNNER" == "dataflow" ]]; then
+  # Verify docker and gcloud commands exist
+  command -v docker
+  docker -v
+  command -v gcloud
+  gcloud --version
+
+  # ensure gcloud is version 186 or above
+  TMPDIR=$(mktemp -d)
+  gcloud_ver=$(gcloud -v | head -1 | awk '{print $4}')
+  if [[ "$gcloud_ver" < "186" ]]
+  then
+    pushd $TMPDIR
+    curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-186.0.0-linux-x86_64.tar.gz --output gcloud.tar.gz
+    tar xf gcloud.tar.gz
+    ./google-cloud-sdk/install.sh --quiet
+    . ./google-cloud-sdk/path.bash.inc
+    popd
+    gcloud components update --quiet || echo 'gcloud components update failed'
+    gcloud -v
+  fi
 
-  ARGS="$ARGS --endpoint=$ENDPOINT --expansion_addr=$EXPANSION_ADDR"
+  # Build the container
+  TAG=$(date +%Y%m%d-%H%M%S)
+  CONTAINER=us.gcr.io/$PROJECT/$USER/beam_go_sdk

Review comment:
       The $USER here is just the standard OS $USER executing the script then? Just confirming something wasn't missed.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] youngoli commented on a change in pull request #13678: [BEAM-11415] Add Dataflow Runner to new Go ValidatesRunner framework.

Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #13678:
URL: https://github.com/apache/beam/pull/13678#discussion_r553011126



##########
File path: sdks/go/test/run_validatesrunner_tests.sh
##########
@@ -16,39 +16,68 @@
 #    limitations under the License.
 
 # This script executes ValidatesRunner tests including launching any additional
-# services needed, such as job services or expansion services. The following
-# runners are supported, and selected via a flag:
+# services needed, such as job services or expansion services. This script
+# should be executed from the root of the Beam repository.
 #
+# The following runners are supported, and selected via a flag:
 # --runner {portable|direct|flink} (default: portable)
 #  Select which runner to execute tests on. This flag also determines which
 #  services to start up and which tests may be skipped.
 #    direct   - Go SDK Direct Runner
 #    portable - (default) Python Portable Runner (aka. Reference Runner or FnAPI Runner)
 #    flink    - Java Flink Runner (local mode)
 #    spark    - Java Spark Runner (local mode)
+#    dataflow - Dataflow Runner
 #
-# --flink_job_server_jar -> Filepath to jar, used if runner is Flink.
-# --spark_job_server_jar -> Filepath to jar, used if runner is Spark.
-# --endpoint -> Replaces jar filepath with existing job server endpoint.
+# General flags:
+#    --timeout -> Timeout for the go test command, on a per-package level.
+#    --endpoint -> An endpoint for an existing job server outside the script.
+#        If present, job server jar flags are ignored.
+#    --expansion_service_jar -> Filepath to jar for expansion service, for
+#        runners that support cross-language.
+#    --expansion_addr -> An endpoint for an existing expansion service outside
+#        the script. If present, expansion_service_jar is ignored.
 #
-# --expansion_service_jar -> Filepath to jar for expansion service.
-# --expansion_addr -> Replaces jar filepath with existing expansion service endpoint.
-#
-# Execute from the root of the repository. This script requires that necessary
-# services can be built from the repository.
+# Runner-specific flags:
+#  Flink
+#    --flink_job_server_jar -> Filepath to jar, used if runner is Flink.
+#  Spark
+#    --spark_job_server_jar -> Filepath to jar, used if runner is Spark.
+#  Dataflow
+#    --dataflow_project -> GCP project to run Dataflow jobs on.
+#    --project -> Same project as dataflow-project, but in URL format, for

Review comment:
       Done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] youngoli commented on pull request #13678: [BEAM-11415] Add Dataflow Runner to new Go ValidatesRunner framework.

Posted by GitBox <gi...@apache.org>.
youngoli commented on pull request #13678:
URL: https://github.com/apache/beam/pull/13678#issuecomment-755737663


   R: @lostluck 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] youngoli commented on a change in pull request #13678: [BEAM-11415] Add Dataflow Runner to new Go ValidatesRunner framework.

Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #13678:
URL: https://github.com/apache/beam/pull/13678#discussion_r553011334



##########
File path: sdks/go/test/run_validatesrunner_tests.sh
##########
@@ -16,39 +16,68 @@
 #    limitations under the License.
 
 # This script executes ValidatesRunner tests including launching any additional
-# services needed, such as job services or expansion services. The following
-# runners are supported, and selected via a flag:
+# services needed, such as job services or expansion services. This script
+# should be executed from the root of the Beam repository.
 #
+# The following runners are supported, and selected via a flag:
 # --runner {portable|direct|flink} (default: portable)
 #  Select which runner to execute tests on. This flag also determines which
 #  services to start up and which tests may be skipped.
 #    direct   - Go SDK Direct Runner
 #    portable - (default) Python Portable Runner (aka. Reference Runner or FnAPI Runner)
 #    flink    - Java Flink Runner (local mode)
 #    spark    - Java Spark Runner (local mode)
+#    dataflow - Dataflow Runner
 #
-# --flink_job_server_jar -> Filepath to jar, used if runner is Flink.
-# --spark_job_server_jar -> Filepath to jar, used if runner is Spark.
-# --endpoint -> Replaces jar filepath with existing job server endpoint.
+# General flags:
+#    --timeout -> Timeout for the go test command, on a per-package level.
+#    --endpoint -> An endpoint for an existing job server outside the script.
+#        If present, job server jar flags are ignored.
+#    --expansion_service_jar -> Filepath to jar for expansion service, for
+#        runners that support cross-language.
+#    --expansion_addr -> An endpoint for an existing expansion service outside
+#        the script. If present, expansion_service_jar is ignored.
 #
-# --expansion_service_jar -> Filepath to jar for expansion service.
-# --expansion_addr -> Replaces jar filepath with existing expansion service endpoint.
-#
-# Execute from the root of the repository. This script requires that necessary
-# services can be built from the repository.
+# Runner-specific flags:
+#  Flink
+#    --flink_job_server_jar -> Filepath to jar, used if runner is Flink.
+#  Spark
+#    --spark_job_server_jar -> Filepath to jar, used if runner is Spark.
+#  Dataflow
+#    --dataflow_project -> GCP project to run Dataflow jobs on.
+#    --project -> Same project as dataflow-project, but in URL format, for
+#        example in the format "us.gcr.io/<project>".
+#    --region -> GCP region to run Dataflow jobs on.
+#    --gcs_location -> GCS URL for storing temporary files for Dataflow jobs.
+#    --dataflow_worker_jar -> The Dataflow worker jar to use when running jobs.
+#        If not specified, the script attempts to retrieve a previously built
+#        jar from the appropriate gradle module, which may not succeed.
 
 set -e
 set -v
 
+# Default runner.
 RUNNER=portable
 
+# Default timeout. This timeout is applied per-package, as tests in different
+# packages are executed in parallel.
+TIMEOUT=1h
+
+# Where to store integration test outputs.
+GCS_LOCATION=gs://clouddfe-danoliveira/temp-storage-go-end-to-end-tests

Review comment:
       Whoops, forgot to change that. These should match the old framework's defaults, changing it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] youngoli merged pull request #13678: [BEAM-11415] Add Dataflow Runner to new Go ValidatesRunner framework.

Posted by GitBox <gi...@apache.org>.
youngoli merged pull request #13678:
URL: https://github.com/apache/beam/pull/13678


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] youngoli commented on a change in pull request #13678: [BEAM-11415] Add Dataflow Runner to new Go ValidatesRunner framework.

Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #13678:
URL: https://github.com/apache/beam/pull/13678#discussion_r553011431



##########
File path: sdks/go/test/run_validatesrunner_tests.sh
##########
@@ -151,11 +222,70 @@ if [[ "$RUNNER" == "flink" || "$RUNNER" == "spark" || "$RUNNER" == "portable" ]]
     java -jar $EXPANSION_SERVICE_JAR $EXPANSION_PORT &
     EXPANSION_PID=$!
   fi
+fi
+
+if [[ "$RUNNER" == "dataflow" ]]; then
+  # Verify docker and gcloud commands exist
+  command -v docker
+  docker -v
+  command -v gcloud
+  gcloud --version
+
+  # ensure gcloud is version 186 or above
+  TMPDIR=$(mktemp -d)
+  gcloud_ver=$(gcloud -v | head -1 | awk '{print $4}')
+  if [[ "$gcloud_ver" < "186" ]]
+  then
+    pushd $TMPDIR
+    curl https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-186.0.0-linux-x86_64.tar.gz --output gcloud.tar.gz
+    tar xf gcloud.tar.gz
+    ./google-cloud-sdk/install.sh --quiet
+    . ./google-cloud-sdk/path.bash.inc
+    popd
+    gcloud components update --quiet || echo 'gcloud components update failed'
+    gcloud -v
+  fi
 
-  ARGS="$ARGS --endpoint=$ENDPOINT --expansion_addr=$EXPANSION_ADDR"
+  # Build the container
+  TAG=$(date +%Y%m%d-%H%M%S)
+  CONTAINER=us.gcr.io/$PROJECT/$USER/beam_go_sdk

Review comment:
       Hmm, looks like it. When I copied the code over I didn't notice that no USER variable was defined. On the other hand, this is just used to generate a directory for the script to store containers, so it's probably unnecessary to have a flag for this unless it causes problems.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org