You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/12 07:13:04 UTC

[GitHub] [beam] TobKed opened a new pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

TobKed opened a new pull request #12541:
URL: https://github.com/apache/beam/pull/12541


   Workflow run with java tests
   
   # Before merging it is required to setup related secrets:
    - `GCP_TESTING_BUCKET` - beam-github-actions-tests
    - `GCP_PROJECT_ID - apache-beam-testing
    - `GCP_REGION` - us-central1
   
    # Before merging it Service Account should have elevated permissions:
    - Storage Admin (roles/storage.admin)
    - Dataflow Admin (roles/dataflow.admin)
   
   # Information about PR
   Example run on my local fork: https://github.com/TobKed/beam/actions/runs/205143020
   
   For tests failing on windows I created JIRA [BEAM-10683](https://issues.apache.org/jira/browse/BEAM-10683). 
   I didn't want to solve them here since I am not so much familiar with java and Beam logic so I didn't want just make tests green, what could hide potentially real issue.
   
   
   
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](htt
 ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_
 Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_P
 ostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website
   --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/b
 eam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   ![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-676077406


   Rebased on the latest master and solved conflicts


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r471498608



##########
File path: CI.md
##########
@@ -75,8 +75,27 @@ run categories. Here is a summary of the run categories with regards of the jobs
 Those jobs often have matrix run strategy which runs several different variations of the jobs
 (with different platform type / Python version to run for example)
 
+### Google Cloud Platform Credentials
+
+Some of the jobs require variables stored as a [GitHub Secrets](https://docs.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets)
+to perform operations on Google Cloud Platform. Currently these jobs are limited to Apache repository only.

Review comment:
       Yes, if variables are set workflow can be fully tested on the fork. It will be much more easier thanks to manual trigger by `workflow_dispatch` event which provide button to do this. Sentence `Currently these jobs are limited to Apache repository only` is not valid since GCP variables check was introduced. I missed that. I will create separate PR for this fix. Thanks for pointing this out.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-675585059


   Run Python PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-679010663


   # [Codecov](https://codecov.io/gh/apache/beam/pull/12541?src=pr&el=h1) Report
   > Merging [#12541](https://codecov.io/gh/apache/beam/pull/12541?src=pr&el=desc) into [master](https://codecov.io/gh/apache/beam/commit/cfa448d121297398312d09c531258a72b413488b?el=desc) will **not change** coverage.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/beam/pull/12541/graphs/tree.svg?width=650&height=150&src=pr&token=qcbbAh8Fj1)](https://codecov.io/gh/apache/beam/pull/12541?src=pr&el=tree)
   
   ```diff
   @@           Coverage Diff           @@
   ##           master   #12541   +/-   ##
   =======================================
     Coverage   34.47%   34.47%           
   =======================================
     Files         684      684           
     Lines       81483    81483           
     Branches     9180     9180           
   =======================================
     Hits        28090    28090           
   - Misses      52965    52972    +7     
   + Partials      428      421    -7     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/12541?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [io/mongodbio\_test.py](https://codecov.io/gh/apache/beam/pull/12541/diff?src=pr&el=tree#diff-aW8vbW9uZ29kYmlvX3Rlc3QucHk=) | `34.95% <0.00%> (ø)` | |
   | [runners/direct/direct\_runner.py](https://codecov.io/gh/apache/beam/pull/12541/diff?src=pr&el=tree#diff-cnVubmVycy9kaXJlY3QvZGlyZWN0X3J1bm5lci5weQ==) | `32.88% <0.00%> (ø)` | |
   | [io/gcp/experimental/spannerio.py](https://codecov.io/gh/apache/beam/pull/12541/diff?src=pr&el=tree#diff-aW8vZ2NwL2V4cGVyaW1lbnRhbC9zcGFubmVyaW8ucHk=) | `34.49% <0.00%> (ø)` | |
   | [typehints/typed\_pipeline\_test.py](https://codecov.io/gh/apache/beam/pull/12541/diff?src=pr&el=tree#diff-dHlwZWhpbnRzL3R5cGVkX3BpcGVsaW5lX3Rlc3QucHk=) | `21.98% <0.00%> (ø)` | |
   | [runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/12541/diff?src=pr&el=tree#diff-cnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `28.45% <0.00%> (ø)` | |
   | [runners/dataflow/dataflow\_runner\_test.py](https://codecov.io/gh/apache/beam/pull/12541/diff?src=pr&el=tree#diff-cnVubmVycy9kYXRhZmxvdy9kYXRhZmxvd19ydW5uZXJfdGVzdC5weQ==) | `25.49% <0.00%> (ø)` | |
   | [runners/dataflow/ptransform\_overrides.py](https://codecov.io/gh/apache/beam/pull/12541/diff?src=pr&el=tree#diff-cnVubmVycy9kYXRhZmxvdy9wdHJhbnNmb3JtX292ZXJyaWRlcy5weQ==) | `18.11% <0.00%> (ø)` | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12541?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12541?src=pr&el=footer). Last update [cfa448d...e164d17](https://codecov.io/gh/apache/beam/pull/12541?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-679009548


   > @tysonjh @tvalentyn thanks for reviews. What are next steps? should I do something more for this PR?
   
   cc: @tysonjh @tvalentyn @aaltay @kamilwu @mwalenia 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r471554964



##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}

Review comment:
       Each workflow run have artifacts connected to it. There is no risk of race condition between workflows artifacts.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-672892299


   Created single PR to infra with all required changes for github secrets: https://issues.apache.org/jira/browse/INFRA-20675


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed edited a comment on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed edited a comment on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-675591490


   @tvalentyn @aaltay @tysonjh  I see that https://issues.apache.org/jira/browse/INFRA-20675 is done and just waits for confirmation.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r472105761



##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/io/DefaultFilenamePolicyTest.java
##########
@@ -28,6 +31,12 @@
 @RunWith(JUnit4.class)
 public class DefaultFilenamePolicyTest {
 
+  @Before
+  public void setup() {
+    // TODO: Fix failing on windows, https://issues.apache.org/jira/browse/BEAM-10683.

Review comment:
       Sure. I created 26 subtasks. Every failing test or testing class have link to relevant JIRA issue now.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-675547993


   Run Python2_PVR_Flink PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r472105761



##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/io/DefaultFilenamePolicyTest.java
##########
@@ -28,6 +31,12 @@
 @RunWith(JUnit4.class)
 public class DefaultFilenamePolicyTest {
 
+  @Before
+  public void setup() {
+    // TODO: Fix failing on windows, https://issues.apache.org/jira/browse/BEAM-10683.

Review comment:
       Sure. I created 26 subtasks for https://issues.apache.org/jira/browse/BEAM-10683
   Every failing test or testing class have link to relevant JIRA issue now.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r471522549



##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}
+          path: sdks/java/core/build/reports/tests/test/**
+      # :sdks:java:harness:test
+      - name: Run :sdks:java:harness:test
+        if: ${{ always() }}
+        run: ./gradlew :sdks:java:harness:test
+      - name: Upload test logs for :sdks:java:harness:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-harness-test-${{ matrix.os }}
+          path: sdks/java/harness/build/reports/tests/test/**
+      # :runners:core-java:test
+      - name: Run :runners:core-java:test
+        if: ${{ always() }}
+        run: ./gradlew :runners:core-java:test
+      - name: Upload test logs for :runners:core-java:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-runners-core-java-test-${{ matrix.os }}
+          path: runners/core-java/build/reports/tests/test/**
+
+  java_wordcount_direct_runner:
+    name: 'Java Wordcount Direct Runner'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Run WordCount Unix
+        shell: bash
+        run: |
+          bash ./gradlew -p examples/ \
+             integrationTest -DintegrationTestPipelineOptions='[
+          	 "--runner=DirectRunner", "--tempRoot=./tmp" ]' \
+             --tests org.apache.beam.examples.WordCountIT \
+             -DintegrationTestRunner=direct
+      - name: Upload test logs
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_wordcount_direct_runner-${{matrix.os}}
+          path: examples/java/build/reports/tests/integrationTest

Review comment:
       I am not sure did I understand. Is question about why there is `examples/java/build/reports/tests/integrationTest` instead of `examples/java/build/reports/tests/integrationTest/**`?
   I it was a question, I will simplify paths for `java_unit_tests` logs paths to avoid unnecessary globstar.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-675591490


   @tvalentyn @aaltay I see that https://issues.apache.org/jira/browse/INFRA-20675 is done and just waits for confirmation.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-675430671


   Thanks @tysonjh . I created seperate JIRAs for failing tests, rebased on the latest master and squashed commits.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r471710564



##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}
+          path: sdks/java/core/build/reports/tests/test/**
+      # :sdks:java:harness:test
+      - name: Run :sdks:java:harness:test
+        if: ${{ always() }}
+        run: ./gradlew :sdks:java:harness:test
+      - name: Upload test logs for :sdks:java:harness:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-harness-test-${{ matrix.os }}
+          path: sdks/java/harness/build/reports/tests/test/**
+      # :runners:core-java:test
+      - name: Run :runners:core-java:test
+        if: ${{ always() }}
+        run: ./gradlew :runners:core-java:test
+      - name: Upload test logs for :runners:core-java:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-runners-core-java-test-${{ matrix.os }}
+          path: runners/core-java/build/reports/tests/test/**
+
+  java_wordcount_direct_runner:
+    name: 'Java Wordcount Direct Runner'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Run WordCount Unix
+        shell: bash
+        run: |

Review comment:
       The reason behind it is to override the default powershell on windows and ensure that gradle will use bash as well. This way there is no need for separate steps for windows and unix systems.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] kamilwu commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-678132113


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] kamilwu commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-678106411


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] potiuk commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r471538073



##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()

Review comment:
       Yeah. This is the right way of doing it. By default all steps in the action have "if: success()" implied. Adding if: always() is the standard way if you want to execute step regardless of the status of previous ones. It is supposed to be executed even if you cancel the flow https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#always 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r471555151



##########
File path: .github/workflows/cancel.yml
##########
@@ -32,3 +35,18 @@ jobs:
         with:
           workflow_id: ${{ env.WORKFLOW_ID }}
           access_token: ${{ github.token }}
+
+  cancel_java_tests:
+    name: 'Cancel Previous Runs of Java Tests'
+    runs-on: ubuntu-latest
+    timeout-minutes: 3
+    steps:
+      - name: Get build_wheels workflow id

Review comment:
       Thanks. Suggestion applied.

##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}
+          path: sdks/java/core/build/reports/tests/test/**
+      # :sdks:java:harness:test
+      - name: Run :sdks:java:harness:test
+        if: ${{ always() }}

Review comment:
       Thanks. Suggestion applied.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-672830626


   R: @aaltay @tvalentyn @mwalenia @damgad 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r471538676



##########
File path: .github/workflows/cancel.yml
##########
@@ -32,3 +35,18 @@ jobs:
         with:
           workflow_id: ${{ env.WORKFLOW_ID }}
           access_token: ${{ github.token }}
+
+  cancel_java_tests:

Review comment:
       I have to add it for python tests suite as well.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-672892447


   Run Python PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r471538676



##########
File path: .github/workflows/cancel.yml
##########
@@ -32,3 +35,18 @@ jobs:
         with:
           workflow_id: ${{ env.WORKFLOW_ID }}
           access_token: ${{ github.token }}
+
+  cancel_java_tests:

Review comment:
       I have to added it for python tests suite as well.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-675548197






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r471532668



##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()

Review comment:
       This [job status check function](https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#job-status-check-functions) provides a convenient way to run step even if previous one fails. Without using it failing test will prevent log upload and execution of the rest of tests.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] potiuk commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r471542050



##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}

Review comment:
       Each run has its own set of artifacts. GithubActions for artifact upload/dowload work for the "current run" only. You can download artifacts for other runs via APIS, but I believe you can't upload them from a different run than current. It even works in the way that you actually cannot see the artifacts via the API before the run is finished unless you are with the current run. This allows to share artifacts between different jobs in the same run, but you cannot use the artifacts between the runs. That effectively makes the artifacts immutable after the run completes  (been there done that)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tysonjh commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
tysonjh commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r470903163



##########
File path: .github/workflows/cancel.yml
##########
@@ -32,3 +35,18 @@ jobs:
         with:
           workflow_id: ${{ env.WORKFLOW_ID }}
           access_token: ${{ github.token }}
+
+  cancel_java_tests:

Review comment:
       Is there an equivalent for python or go tests somewhere?

##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()

Review comment:
       If `if: always()` is always true, is this condition really needed? (here and elsewhere in this workflow)

##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}
+          path: sdks/java/core/build/reports/tests/test/**
+      # :sdks:java:harness:test
+      - name: Run :sdks:java:harness:test
+        if: ${{ always() }}

Review comment:
       ```suggestion
           if: always()
   ```

##########
File path: .github/workflows/cancel.yml
##########
@@ -32,3 +35,18 @@ jobs:
         with:
           workflow_id: ${{ env.WORKFLOW_ID }}
           access_token: ${{ github.token }}
+
+  cancel_java_tests:
+    name: 'Cancel Previous Runs of Java Tests'
+    runs-on: ubuntu-latest
+    timeout-minutes: 3
+    steps:
+      - name: Get build_wheels workflow id

Review comment:
       ```suggestion
         - name: Get java_tests workflow id
   ```

##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}

Review comment:
       Is there a risk that concurrently running workflows could race to write this artifact since it doesn't contain any globally unique information (e.g. date)? Or are artifact names unique to each individual workflow run?

##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}
+          path: sdks/java/core/build/reports/tests/test/**
+      # :sdks:java:harness:test
+      - name: Run :sdks:java:harness:test
+        if: ${{ always() }}
+        run: ./gradlew :sdks:java:harness:test
+      - name: Upload test logs for :sdks:java:harness:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-harness-test-${{ matrix.os }}
+          path: sdks/java/harness/build/reports/tests/test/**
+      # :runners:core-java:test
+      - name: Run :runners:core-java:test
+        if: ${{ always() }}
+        run: ./gradlew :runners:core-java:test
+      - name: Upload test logs for :runners:core-java:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-runners-core-java-test-${{ matrix.os }}
+          path: runners/core-java/build/reports/tests/test/**
+
+  java_wordcount_direct_runner:
+    name: 'Java Wordcount Direct Runner'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Run WordCount Unix
+        shell: bash
+        run: |
+          bash ./gradlew -p examples/ \
+             integrationTest -DintegrationTestPipelineOptions='[
+          	 "--runner=DirectRunner", "--tempRoot=./tmp" ]' \
+             --tests org.apache.beam.examples.WordCountIT \
+             -DintegrationTestRunner=direct
+      - name: Upload test logs
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_wordcount_direct_runner-${{matrix.os}}
+          path: examples/java/build/reports/tests/integrationTest

Review comment:
       Why upload the whole directory here instead of a globstar? Just curious.

##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}
+          path: sdks/java/core/build/reports/tests/test/**
+      # :sdks:java:harness:test
+      - name: Run :sdks:java:harness:test
+        if: ${{ always() }}
+        run: ./gradlew :sdks:java:harness:test
+      - name: Upload test logs for :sdks:java:harness:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-harness-test-${{ matrix.os }}
+          path: sdks/java/harness/build/reports/tests/test/**
+      # :runners:core-java:test
+      - name: Run :runners:core-java:test
+        if: ${{ always() }}
+        run: ./gradlew :runners:core-java:test
+      - name: Upload test logs for :runners:core-java:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-runners-core-java-test-${{ matrix.os }}
+          path: runners/core-java/build/reports/tests/test/**
+
+  java_wordcount_direct_runner:
+    name: 'Java Wordcount Direct Runner'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Run WordCount Unix
+        shell: bash
+        run: |

Review comment:
       Why is a shell used in some jobs but not others (specified here as a config and used in the run config below)? Just curious if it is needed for some reason here specifically since it is an integration test.

##########
File path: CI.md
##########
@@ -75,8 +75,27 @@ run categories. Here is a summary of the run categories with regards of the jobs
 Those jobs often have matrix run strategy which runs several different variations of the jobs
 (with different platform type / Python version to run for example)
 
+### Google Cloud Platform Credentials
+
+Some of the jobs require variables stored as a [GitHub Secrets](https://docs.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets)
+to perform operations on Google Cloud Platform. Currently these jobs are limited to Apache repository only.

Review comment:
       A user can set these secrets themselves though to test on their own fork?

##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/io/DefaultFilenamePolicyTest.java
##########
@@ -28,6 +31,12 @@
 @RunWith(JUnit4.class)
 public class DefaultFilenamePolicyTest {
 
+  @Before
+  public void setup() {
+    // TODO: Fix failing on windows, https://issues.apache.org/jira/browse/BEAM-10683.

Review comment:
       Can you create a Jira issue per failing test suite as subtasks of BEAM-10683 please? Having a list of failing test suites will be easier for tracking and hopefully encourage people to chip away at it instead of one large issue for them all.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-678168160


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-675724362


    Ack, will defer to @tysonjh re: next steps. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] kamilwu commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-679114434


   It looks like all comments have been addressed, so I'm going to merge it now. Thanks!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-677628969


   @tysonjh @tvalentyn thanks for reviews. What are next steps? should I do something more for this PR?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-676200226


   Rebased and fixed GCP variables checks after merging https://github.com/apache/beam/pull/12513


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aaltay commented on pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12541:
URL: https://github.com/apache/beam/pull/12541#issuecomment-673020826


   R: @tysonjh / @kennknowles - Could one of you review this please?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tysonjh commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
tysonjh commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r471837656



##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+  workflow_dispatch:
+    inputs:
+      runDataflow:
+        description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+        default: false
+  schedule:
+    - cron: '10 2 * * *'
+  push:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+  pull_request:
+    branches: ['master', 'release-*']
+    tags: 'v*'
+    paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+  check_gcp_variables:
+    timeout-minutes: 5
+    name: "Check GCP variables set"
+    runs-on: ubuntu-latest
+    outputs:
+      gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+    steps:
+      - uses: actions/checkout@v2
+      - name: "Check are GCP variables set"
+        run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+        id: check_gcp_variables
+        env:
+          GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+          GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+          GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+          GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      # :sdks:java:core:test
+      - name: Run :sdks:java:core:test
+        run: ./gradlew :sdks:java:core:test
+      - name: Upload test logs for :sdks:java:core:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}
+          path: sdks/java/core/build/reports/tests/test/**
+      # :sdks:java:harness:test
+      - name: Run :sdks:java:harness:test
+        if: ${{ always() }}
+        run: ./gradlew :sdks:java:harness:test
+      - name: Upload test logs for :sdks:java:harness:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-sdks-java-harness-test-${{ matrix.os }}
+          path: sdks/java/harness/build/reports/tests/test/**
+      # :runners:core-java:test
+      - name: Run :runners:core-java:test
+        if: ${{ always() }}
+        run: ./gradlew :runners:core-java:test
+      - name: Upload test logs for :runners:core-java:test
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_unit_tests-runners-core-java-test-${{ matrix.os }}
+          path: runners/core-java/build/reports/tests/test/**
+
+  java_wordcount_direct_runner:
+    name: 'Java Wordcount Direct Runner'
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Run WordCount Unix
+        shell: bash
+        run: |
+          bash ./gradlew -p examples/ \
+             integrationTest -DintegrationTestPipelineOptions='[
+          	 "--runner=DirectRunner", "--tempRoot=./tmp" ]' \
+             --tests org.apache.beam.examples.WordCountIT \
+             -DintegrationTestRunner=direct
+      - name: Upload test logs
+        uses: actions/upload-artifact@v2
+        if: always()
+        with:
+          name: java_wordcount_direct_runner-${{matrix.os}}
+          path: examples/java/build/reports/tests/integrationTest

Review comment:
       Yes, this was the question. If there is no reason to differentiate, a consistent path is preferred. Thanks!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] TobKed commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r471509769



##########
File path: CI.md
##########
@@ -75,8 +75,27 @@ run categories. Here is a summary of the run categories with regards of the jobs
 Those jobs often have matrix run strategy which runs several different variations of the jobs
 (with different platform type / Python version to run for example)
 
+### Google Cloud Platform Credentials
+
+Some of the jobs require variables stored as a [GitHub Secrets](https://docs.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets)
+to perform operations on Google Cloud Platform. Currently these jobs are limited to Apache repository only.

Review comment:
       https://github.com/apache/beam/pull/12598




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] kamilwu merged pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac

Posted by GitBox <gi...@apache.org>.
kamilwu merged pull request #12541:
URL: https://github.com/apache/beam/pull/12541


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org