You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/07/20 13:26:45 UTC

[GitHub] [beam] TobKed opened a new pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

TobKed opened a new pull request #12318:
URL: https://github.com/apache/beam/pull/12318


   Cross-platform test suite to be run during nightly build and on push to master and release branches.
   
   Requires adding DataFlow admin permissions to GCP Service Account before merging
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](htt
 ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_
 Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_P
 ostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website
   --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TobKed commented on a change in pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r459376217



##########
File path: .github/workflows/build_wheels.yml
##########
@@ -193,29 +196,232 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${{ env.GCP_PATH }}
+        gsutil cp -a public-read github_action_info ${{ env.GCP_WHEELS_STAGING_PATH }}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_PATH }}
+      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_WHEELS_STAGING_PATH }}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
-    needs: upload_wheels_to_gcs
+    needs: upload_python_sdk_wheels_to_gcs
     runs-on: ubuntu-latest
-    if: github.repository_owner == 'apache' && github.event_name != 'pull_request'
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
     steps:
     - name: Authenticate on GCP
       uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
       with:
         service_account_email: ${{ secrets.GCP_SA_EMAIL }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${{ env.GCP_PATH }}*"
+      run: gsutil ls "${{ env.GCP_WHEELS_STAGING_PATH }}*"
+
+  python_unit_tests:
+    name: 'Python Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'

Review comment:
       > Question: Do we want to run on this on every push? This might have quite a bit of overlap with precommits and might be slow.
   
   push event is limited only to master and release-* branches so I think it shouldn't overlap with precommits.
   
   > Also, are actions workflow subject to any kind of quota? Are we at risk of exceeding it? Can we monitor it somehow?
   
    Quoting docs: "GitHub Actions usage is free for public repositories." so I suppose there is no quota for the time. There are usage limits for concurrent jobs but I think it should be not a problem since Apache has Enterprise plan (I couldn't find official info anywhere)
   
   docs:
   https://docs.github.com/en/actions/getting-started-with-github-actions/about-github-actions#usage-limits
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] aaltay commented on a change in pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r458272138



##########
File path: .github/workflows/build_wheels.yml
##########
@@ -193,29 +196,232 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${{ env.GCP_PATH }}
+        gsutil cp -a public-read github_action_info ${{ env.GCP_WHEELS_STAGING_PATH }}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_PATH }}
+      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_WHEELS_STAGING_PATH }}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
-    needs: upload_wheels_to_gcs
+    needs: upload_python_sdk_wheels_to_gcs
     runs-on: ubuntu-latest
-    if: github.repository_owner == 'apache' && github.event_name != 'pull_request'
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
     steps:
     - name: Authenticate on GCP
       uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
       with:
         service_account_email: ${{ secrets.GCP_SA_EMAIL }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${{ env.GCP_PATH }}*"
+      run: gsutil ls "${{ env.GCP_WHEELS_STAGING_PATH }}*"
+
+  python_unit_tests:
+    name: 'Python Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        params: [
+          {"py_ver": "3.5", "tox_env": "py35"},
+          {"py_ver": "3.6", "tox_env": "py36"},
+          {"py_ver": "3.7", "tox_env": "py37"},
+        ]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Install python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.params.py_ver }}
+      - name: Get build dependencies
+        working-directory: ./sdks/python
+        run: python -m pip install -r build-requirements.txt
+      - name: Install tox
+        run: pip install tox
+      - name: Run tests basic unix
+        if: startsWith(matrix.os, 'ubuntu') || startsWith(matrix.os, 'macos')
+        working-directory: ./sdks/python
+        run: tox -c tox.ini -e ${{ matrix.params.tox_env }}
+      - name: Run tests basic windows
+        if: startsWith(matrix.os, 'windows') && (matrix.params.py_ver != '2.7')
+        working-directory: ./sdks/python
+        run: tox -c tox.ini -e ${{ matrix.params.tox_env }}-win
+
+  python_wordcount_direct_runner:
+    name: 'Python Wordcount Direct Runner'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python: [3.5, 3.6, 3.7]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Install python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python }}
+      - name: Get build dependencies
+        working-directory: ./sdks/python
+        run: python -m pip install -r build-requirements.txt
+      - name: Install requirements
+        working-directory: ./sdks/python
+        run: pip install -e .
+      - name: Run WordCount
+        working-directory: ./sdks/python
+        run: python -m apache_beam.examples.wordcount --input MANIFEST.in --output counts
+
+  python_wordcount_dataflow:
+    name: 'Python Wordcount Dataflow'
+    needs: build_python_sdk_source
+    runs-on: ${{ matrix.os }}
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python: [3.5, 3.6, 3.7]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Install python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python }}
+      - name: Download source from artifacts
+        uses: actions/download-artifact@v2
+        with:
+          name: source_gztar_zip
+          path: apache-beam-source
+      - name: Authenticate on GCP
+        uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+        with:
+          service_account_email: ${{ secrets.GCP_SA_EMAIL }}
+          service_account_key: ${{ secrets.GCP_SA_KEY }}
+          export_default_credentials: true
+      - name: Get build dependencies
+        working-directory: ./sdks/python
+        run: python -m pip install -r build-requirements.txt
+      - name: Install requirements
+        working-directory: ./sdks/python
+        run: pip install -e ".[gcp]"
+      - name: Run WordCount
+        working-directory: ./sdks/python
+        shell: bash
+        run: |
+          python -m apache_beam.examples.wordcount \
+            --input gs://dataflow-samples/shakespeare/kinglear.txt \
+            --output gs://${{ env.GCP_BUCKET }}/counts \
+            --runner DataflowRunner \
+            --project ${{ env.GCP_PROJECT_ID }} \
+            --region ${{ env.GCP_REGION }} \
+            --temp_location gs://${{ env.GCP_BUCKET }}/tmp/ \
+            --sdk_location "../../apache-beam-source/$(ls ../../apache-beam-source | grep '^apache-beam.*zip$')"
+
+  java_unit_tests:
+    name: 'Java Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Run tests
+        run: |
+          ./gradlew :sdks:java:core:test
+          ./gradlew :sdks:java:harness:test
+          ./gradlew :runners:core-java:test
+
+  java_wordcount_direct_runner:
+    name: 'Java Wordcount Direct Runner'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Get the WordCount Code
+        shell: bash
+        run: |
+          mvn archetype:generate \
+            -DarchetypeGroupId=org.apache.beam \
+            -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+            -DarchetypeVersion=2.22.0 \
+            -DgroupId=org.example \
+            -DartifactId=word-count-beam \
+            -Dversion="0.1" \
+            -Dpackage=org.apache.beam.examples \
+            -DinteractiveMode=false
+      - name: Run WordCount
+        working-directory: ./word-count-beam
+        shell: bash
+        run: |
+          mvn compile exec:java \
+            -Dexec.mainClass=org.apache.beam.examples.WordCount \
+            -Dexec.args="--inputFile=pom.xml --output=counts" \
+            -Pdirect-runner
+
+  java_wordcount_dataflow:
+    name: 'Java Wordcount Dataflow'
+    runs-on: ${{ matrix.os }}
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Authenticate on GCP
+        uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+        with:
+          service_account_email: ${{ secrets.GCP_SA_EMAIL }}
+          service_account_key: ${{ secrets.GCP_SA_KEY }}
+          export_default_credentials: true
+      - name: Get the WordCount Code
+        shell: bash
+        run: |
+          mvn archetype:generate \
+            -DarchetypeGroupId=org.apache.beam \
+            -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
+            -DarchetypeVersion=2.22.0 \

Review comment:
       Should this version match what is at head?

##########
File path: .github/workflows/build_wheels.yml
##########
@@ -193,29 +196,232 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${{ env.GCP_PATH }}
+        gsutil cp -a public-read github_action_info ${{ env.GCP_WHEELS_STAGING_PATH }}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_PATH }}
+      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_WHEELS_STAGING_PATH }}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
-    needs: upload_wheels_to_gcs
+    needs: upload_python_sdk_wheels_to_gcs
     runs-on: ubuntu-latest
-    if: github.repository_owner == 'apache' && github.event_name != 'pull_request'
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
     steps:
     - name: Authenticate on GCP
       uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
       with:
         service_account_email: ${{ secrets.GCP_SA_EMAIL }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${{ env.GCP_PATH }}*"
+      run: gsutil ls "${{ env.GCP_WHEELS_STAGING_PATH }}*"
+
+  python_unit_tests:
+    name: 'Python Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        params: [
+          {"py_ver": "3.5", "tox_env": "py35"},
+          {"py_ver": "3.6", "tox_env": "py36"},
+          {"py_ver": "3.7", "tox_env": "py37"},
+        ]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Install python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.params.py_ver }}
+      - name: Get build dependencies
+        working-directory: ./sdks/python
+        run: python -m pip install -r build-requirements.txt
+      - name: Install tox
+        run: pip install tox
+      - name: Run tests basic unix
+        if: startsWith(matrix.os, 'ubuntu') || startsWith(matrix.os, 'macos')
+        working-directory: ./sdks/python
+        run: tox -c tox.ini -e ${{ matrix.params.tox_env }}
+      - name: Run tests basic windows
+        if: startsWith(matrix.os, 'windows') && (matrix.params.py_ver != '2.7')

Review comment:
       py_ver 2.7 is not in the list of params. Do we need that check?

##########
File path: .github/workflows/build_wheels.yml
##########
@@ -193,29 +196,232 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${{ env.GCP_PATH }}
+        gsutil cp -a public-read github_action_info ${{ env.GCP_WHEELS_STAGING_PATH }}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_PATH }}
+      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_WHEELS_STAGING_PATH }}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
-    needs: upload_wheels_to_gcs
+    needs: upload_python_sdk_wheels_to_gcs
     runs-on: ubuntu-latest
-    if: github.repository_owner == 'apache' && github.event_name != 'pull_request'
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
     steps:
     - name: Authenticate on GCP
       uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
       with:
         service_account_email: ${{ secrets.GCP_SA_EMAIL }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${{ env.GCP_PATH }}*"
+      run: gsutil ls "${{ env.GCP_WHEELS_STAGING_PATH }}*"
+
+  python_unit_tests:
+    name: 'Python Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        params: [
+          {"py_ver": "3.5", "tox_env": "py35"},

Review comment:
       Is it possible to have the [3.5, 3.6, 3.7] list only in one place?

##########
File path: .github/workflows/build_wheels.yml
##########
@@ -193,29 +196,232 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${{ env.GCP_PATH }}
+        gsutil cp -a public-read github_action_info ${{ env.GCP_WHEELS_STAGING_PATH }}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_PATH }}
+      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_WHEELS_STAGING_PATH }}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
-    needs: upload_wheels_to_gcs
+    needs: upload_python_sdk_wheels_to_gcs
     runs-on: ubuntu-latest
-    if: github.repository_owner == 'apache' && github.event_name != 'pull_request'
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
     steps:
     - name: Authenticate on GCP
       uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
       with:
         service_account_email: ${{ secrets.GCP_SA_EMAIL }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${{ env.GCP_PATH }}*"
+      run: gsutil ls "${{ env.GCP_WHEELS_STAGING_PATH }}*"
+
+  python_unit_tests:
+    name: 'Python Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        params: [
+          {"py_ver": "3.5", "tox_env": "py35"},
+          {"py_ver": "3.6", "tox_env": "py36"},
+          {"py_ver": "3.7", "tox_env": "py37"},

Review comment:
       @tvalentyn - Should we add 3.8?

##########
File path: .github/workflows/build_wheels.yml
##########
@@ -193,29 +196,232 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${{ env.GCP_PATH }}
+        gsutil cp -a public-read github_action_info ${{ env.GCP_WHEELS_STAGING_PATH }}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_PATH }}
+      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_WHEELS_STAGING_PATH }}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
-    needs: upload_wheels_to_gcs
+    needs: upload_python_sdk_wheels_to_gcs
     runs-on: ubuntu-latest
-    if: github.repository_owner == 'apache' && github.event_name != 'pull_request'
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
     steps:
     - name: Authenticate on GCP
       uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
       with:
         service_account_email: ${{ secrets.GCP_SA_EMAIL }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${{ env.GCP_PATH }}*"
+      run: gsutil ls "${{ env.GCP_WHEELS_STAGING_PATH }}*"
+
+  python_unit_tests:
+    name: 'Python Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'

Review comment:
       Question: Do we want to run on this on every push? This might have quite a bit of overlap with precommits and might be slow.
   
   /cc @tvalentyn 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] tvalentyn commented on a change in pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r458253905



##########
File path: sdks/python/apache_beam/runners/portability/portable_runner_test.py
##########
@@ -284,6 +284,7 @@ def create_options(self):
     return options
 
 
+@pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")

Review comment:
       Can we add a TODO(BEAM-....) and include an error that happens in the Jira? Does the error happen on all Python versions? 
   Similarly for other errors.

##########
File path: sdks/python/apache_beam/io/parquetio_test.py
##########
@@ -296,7 +296,9 @@ def test_sink_transform_int96(self):
               path, self.SCHEMA96, num_shards=1, shard_name_template='')
 
   def test_sink_transform(self):
-    with tempfile.NamedTemporaryFile() as dst:
+    dst = tempfile.NamedTemporaryFile()

Review comment:
       +1, this pattern (here and elsewhere) looks strange to me.

##########
File path: sdks/python/apache_beam/runners/worker/log_handler_test.py
##########
@@ -87,7 +87,8 @@ def _verify_fn_log_handler(self, num_log_entries):
         self.assertEqual(
             '%s: %s' % (msg, num_received_log_entries), log_entry.message)
         self.assertTrue(
-            re.match(r'.*/log_handler_test.py:\d+', log_entry.log_location),
+            re.match(
+                r'.*(/|\\)log_handler_test.py:\d+', log_entry.log_location),

Review comment:
       Use `os.path.sep`.

##########
File path: sdks/python/apache_beam/runners/portability/stager_test.py
##########
@@ -167,6 +167,9 @@ def test_no_main_session(self):
 
   # xdist adds unpicklable modules to the main session.
   @pytest.mark.no_xdist
+  @pytest.mark.skip(
+      sys.platform == 'win32',
+      "Windows raise TypeError: can't pickle zipimport.zipimporter objects")

Review comment:
       Is there a bug in dill or cpython pickle that tracks this problem? Does this fail on all Python versions?

##########
File path: sdks/python/apache_beam/testing/datatype_inference_test.py
##########
@@ -174,6 +175,9 @@ def test_infer_typehints_schema(self, _, data, schema):
   @parameterized.expand([(d["name"], d["data"], d["pyarrow_schema"])
                          for d in TEST_DATA])
   @unittest.skipIf(pa is None, "PyArrow is not installed")
+  @unittest.skipIf(
+      sys.platform == "win32",
+      "numpy array dtype is coming as int32 by default in a windows 64 bit")

Review comment:
       How does this test fail? We need a JIRA so that we understand the implications of this for the users? Does this fail on all Python versions?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TobKed commented on a change in pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r462965733



##########
File path: .github/workflows/build_wheels.yml
##########
@@ -193,29 +196,232 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${{ env.GCP_PATH }}
+        gsutil cp -a public-read github_action_info ${{ env.GCP_WHEELS_STAGING_PATH }}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_PATH }}
+      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_WHEELS_STAGING_PATH }}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
-    needs: upload_wheels_to_gcs
+    needs: upload_python_sdk_wheels_to_gcs
     runs-on: ubuntu-latest
-    if: github.repository_owner == 'apache' && github.event_name != 'pull_request'
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
     steps:
     - name: Authenticate on GCP
       uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
       with:
         service_account_email: ${{ secrets.GCP_SA_EMAIL }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${{ env.GCP_PATH }}*"
+      run: gsutil ls "${{ env.GCP_WHEELS_STAGING_PATH }}*"
+
+  python_unit_tests:
+    name: 'Python Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        params: [
+          {"py_ver": "3.5", "tox_env": "py35"},
+          {"py_ver": "3.6", "tox_env": "py36"},
+          {"py_ver": "3.7", "tox_env": "py37"},
+        ]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+      - name: Install python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.params.py_ver }}
+      - name: Get build dependencies
+        working-directory: ./sdks/python
+        run: python -m pip install -r build-requirements.txt
+      - name: Install tox
+        run: pip install tox
+      - name: Run tests basic unix
+        if: startsWith(matrix.os, 'ubuntu') || startsWith(matrix.os, 'macos')
+        working-directory: ./sdks/python
+        run: tox -c tox.ini -e ${{ matrix.params.tox_env }}
+      - name: Run tests basic windows
+        if: startsWith(matrix.os, 'windows') && (matrix.params.py_ver != '2.7')

Review comment:
       Good point. We don't need it here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TobKed commented on pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12318:
URL: https://github.com/apache/beam/pull/12318#issuecomment-661139730


   Fails on MacOS runner are related to the GitHub virtual environments problems: https://github.com/actions/virtual-environments/issues/841


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] tvalentyn commented on a change in pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r458494744



##########
File path: .github/workflows/build_wheels.yml
##########
@@ -193,29 +196,232 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${{ env.GCP_PATH }}
+        gsutil cp -a public-read github_action_info ${{ env.GCP_WHEELS_STAGING_PATH }}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_PATH }}
+      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_WHEELS_STAGING_PATH }}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
-    needs: upload_wheels_to_gcs
+    needs: upload_python_sdk_wheels_to_gcs
     runs-on: ubuntu-latest
-    if: github.repository_owner == 'apache' && github.event_name != 'pull_request'
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
     steps:
     - name: Authenticate on GCP
       uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
       with:
         service_account_email: ${{ secrets.GCP_SA_EMAIL }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${{ env.GCP_PATH }}*"
+      run: gsutil ls "${{ env.GCP_WHEELS_STAGING_PATH }}*"
+
+  python_unit_tests:
+    name: 'Python Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        params: [
+          {"py_ver": "3.5", "tox_env": "py35"},
+          {"py_ver": "3.6", "tox_env": "py36"},
+          {"py_ver": "3.7", "tox_env": "py37"},

Review comment:
       Yes




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TobKed commented on a change in pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r458100870



##########
File path: .github/workflows/ci.yml
##########
@@ -0,0 +1,442 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: CI

Review comment:
       For easier review I made separate steps for changes and renaming. In the first commit you will see added jobs.
   I renamed workflow filename and title since it will be more general CI workflow, building wheels are part of it of course, but keeping old name IMHO would be slightly misleading.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] aaltay commented on a change in pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r457755302



##########
File path: sdks/python/tox.ini
##########
@@ -33,6 +33,7 @@ extras = test
 whitelist_externals =
   false
   time
+  bash

Review comment:
       If we are running this in windows, does this mean, bash, time etc. needs to be available in the windows installation? Are they available in the github provided image?

##########
File path: sdks/python/apache_beam/io/parquetio_test.py
##########
@@ -296,7 +296,9 @@ def test_sink_transform_int96(self):
               path, self.SCHEMA96, num_shards=1, shard_name_template='')
 
   def test_sink_transform(self):
-    with tempfile.NamedTemporaryFile() as dst:
+    dst = tempfile.NamedTemporaryFile()

Review comment:
       What is the reason for this change? (Is it because temp files cannot be re-opened in Windows?)
   
   After dst.close() the file could be deleted anytime and that will make the test flaky. You can use tempfile.mkstemp() instead, which moves the cleanup requirement to the caller.

##########
File path: .github/workflows/ci.yml
##########
@@ -0,0 +1,442 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: CI

Review comment:
       Is this identical to `build_wheels.yml`? What needs to be reviewed? why the name change?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] tvalentyn commented on a change in pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r458494869



##########
File path: .github/workflows/build_wheels.yml
##########
@@ -193,29 +196,232 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${{ env.GCP_PATH }}
+        gsutil cp -a public-read github_action_info ${{ env.GCP_WHEELS_STAGING_PATH }}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_PATH }}
+      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_WHEELS_STAGING_PATH }}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
-    needs: upload_wheels_to_gcs
+    needs: upload_python_sdk_wheels_to_gcs
     runs-on: ubuntu-latest
-    if: github.repository_owner == 'apache' && github.event_name != 'pull_request'
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
     steps:
     - name: Authenticate on GCP
       uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
       with:
         service_account_email: ${{ secrets.GCP_SA_EMAIL }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${{ env.GCP_PATH }}*"
+      run: gsutil ls "${{ env.GCP_WHEELS_STAGING_PATH }}*"
+
+  python_unit_tests:
+    name: 'Python Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'

Review comment:
       Also, are actions workflow subject to any kind of quota? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TobKed commented on a change in pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r459370952



##########
File path: .github/workflows/build_wheels.yml
##########
@@ -193,29 +196,232 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${{ env.GCP_PATH }}
+        gsutil cp -a public-read github_action_info ${{ env.GCP_WHEELS_STAGING_PATH }}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_PATH }}
+      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_WHEELS_STAGING_PATH }}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
-    needs: upload_wheels_to_gcs
+    needs: upload_python_sdk_wheels_to_gcs
     runs-on: ubuntu-latest
-    if: github.repository_owner == 'apache' && github.event_name != 'pull_request'
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
     steps:
     - name: Authenticate on GCP
       uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
       with:
         service_account_email: ${{ secrets.GCP_SA_EMAIL }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${{ env.GCP_PATH }}*"
+      run: gsutil ls "${{ env.GCP_WHEELS_STAGING_PATH }}*"
+
+  python_unit_tests:
+    name: 'Python Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        params: [
+          {"py_ver": "3.5", "tox_env": "py35"},
+          {"py_ver": "3.6", "tox_env": "py36"},
+          {"py_ver": "3.7", "tox_env": "py37"},

Review comment:
       I checked running different version of python in one job but time saved by that is very small (~30min vs ~26min) but is much more harder to figure out what went wrong in long logs:
   https://github.com/TobKed/beam/runs/901872919?check_suite_focus=true
   Jobs will run in parallel manner (if enough runners available at the moment)
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on a change in pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r497125141



##########
File path: sdks/python/apache_beam/runners/portability/stager_test.py
##########
@@ -167,6 +167,9 @@ def test_no_main_session(self):
 
   # xdist adds unpicklable modules to the main session.
   @pytest.mark.no_xdist
+  @pytest.mark.skip(
+      sys.platform == 'win32',
+      "Windows raise TypeError: can't pickle zipimport.zipimporter objects")

Review comment:
       Curious what happened to this skip. We're seeing flakes like this quite often BEAM-10987




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on a change in pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r504333910



##########
File path: sdks/python/apache_beam/runners/portability/stager_test.py
##########
@@ -167,6 +167,9 @@ def test_no_main_session(self):
 
   # xdist adds unpicklable modules to the main session.
   @pytest.mark.no_xdist
+  @pytest.mark.skip(
+      sys.platform == 'win32',
+      "Windows raise TypeError: can't pickle zipimport.zipimporter objects")

Review comment:
       Put up a PR to add it back in https://github.com/apache/beam/pull/13099




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] tvalentyn commented on a change in pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r458495203



##########
File path: .github/workflows/build_wheels.yml
##########
@@ -193,29 +196,232 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${{ env.GCP_PATH }}
+        gsutil cp -a public-read github_action_info ${{ env.GCP_WHEELS_STAGING_PATH }}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_PATH }}
+      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_WHEELS_STAGING_PATH }}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
-    needs: upload_wheels_to_gcs
+    needs: upload_python_sdk_wheels_to_gcs
     runs-on: ubuntu-latest
-    if: github.repository_owner == 'apache' && github.event_name != 'pull_request'
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
     steps:
     - name: Authenticate on GCP
       uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
       with:
         service_account_email: ${{ secrets.GCP_SA_EMAIL }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${{ env.GCP_PATH }}*"
+      run: gsutil ls "${{ env.GCP_WHEELS_STAGING_PATH }}*"
+
+  python_unit_tests:
+    name: 'Python Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+        params: [
+          {"py_ver": "3.5", "tox_env": "py35"},
+          {"py_ver": "3.6", "tox_env": "py36"},
+          {"py_ver": "3.7", "tox_env": "py37"},

Review comment:
       Do all tests against all versions run in parallel?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TobKed commented on pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12318:
URL: https://github.com/apache/beam/pull/12318#issuecomment-662892744


   > Is 'cross-platform' test suite a 'windows' or there are some cross-platform aspects in the suites that we are adding?
   
   @tvalentyn I renamed to follow Jira: `Add test suites to run Beam tests on Linux/Windows/Mac platforms`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TobKed commented on pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12318:
URL: https://github.com/apache/beam/pull/12318#issuecomment-661145340


   R: @damgad @kamilwu @mwalenia @aaltay @tysonjh 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TobKed commented on a change in pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r497319272



##########
File path: sdks/python/apache_beam/runners/portability/stager_test.py
##########
@@ -167,6 +167,9 @@ def test_no_main_session(self):
 
   # xdist adds unpicklable modules to the main session.
   @pytest.mark.no_xdist
+  @pytest.mark.skip(
+      sys.platform == 'win32',
+      "Windows raise TypeError: can't pickle zipimport.zipimporter objects")

Review comment:
       I see it this skip was not included in https://github.com/apache/beam/pull/12452.
   As far i remember after some rebasings and further work on the tests it started to work and for many test runs it didn't fail so I removed this skip.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TobKed closed pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

Posted by GitBox <gi...@apache.org>.
TobKed closed pull request #12318:
URL: https://github.com/apache/beam/pull/12318


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] tvalentyn commented on a change in pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

Posted by GitBox <gi...@apache.org>.
tvalentyn commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r458494869



##########
File path: .github/workflows/build_wheels.yml
##########
@@ -193,29 +196,232 @@ jobs:
         GITHUB_BASE_REF=$GITHUB_BASE_REF
         EOF
         echo $(cat github_action_info)
-        gsutil cp -a public-read github_action_info ${{ env.GCP_PATH }}
+        gsutil cp -a public-read github_action_info ${{ env.GCP_WHEELS_STAGING_PATH }}
     - name: Upload GitHub event file to GCS bucket
-      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_PATH }}
+      run: gsutil cp -a public-read ${GITHUB_EVENT_PATH} ${{ env.GCP_WHEELS_STAGING_PATH }}
 
   list_files_on_gcs:
     name: List files on Google Cloud Storage Bucket
-    needs: upload_wheels_to_gcs
+    needs: upload_python_sdk_wheels_to_gcs
     runs-on: ubuntu-latest
-    if: github.repository_owner == 'apache' && github.event_name != 'pull_request'
+    if: github.repository_owner == 'apache' && (github.event_name == 'push' || github.event_name == 'schedule')
     steps:
     - name: Authenticate on GCP
       uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
       with:
         service_account_email: ${{ secrets.GCP_SA_EMAIL }}
         service_account_key: ${{ secrets.GCP_SA_KEY }}
     - name: List file on Google Cloud Storage Bucket
-      run: gsutil ls "${{ env.GCP_PATH }}*"
+      run: gsutil ls "${{ env.GCP_WHEELS_STAGING_PATH }}*"
+
+  python_unit_tests:
+    name: 'Python Unit Tests'
+    runs-on: ${{ matrix.os }}
+    if: github.event_name == 'push' || github.event_name == 'schedule'

Review comment:
       Also, are actions workflow subject to any kind of quota? Are we at risk of exceeding it? Can we monitor it somehow?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TobKed commented on a change in pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r458094963



##########
File path: sdks/python/tox.ini
##########
@@ -33,6 +33,7 @@ extras = test
 whitelist_externals =
   false
   time
+  bash

Review comment:
       I added bash to whitelist to get rid of warning messages. It is raised because I had to explicitly use bash in *-win tox testenvs. Bash is available on the github runners:
   
   > When specifying a bash shell on Windows, the bash shell included with Git for Windows is  used.
   
   https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#using-a-specific-shell
   
   false and time command are available in mentioned bash shell on windows




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] tvalentyn commented on pull request #12318: [BEAM-10538] Add cross-platform test suite on GitHub Actions

Posted by GitBox <gi...@apache.org>.
tvalentyn commented on pull request #12318:
URL: https://github.com/apache/beam/pull/12318#issuecomment-661997835


   Is 'cross-platform' test suite a 'windows' or there are some cross-platform aspects in the suites that we are adding?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TobKed commented on pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12318:
URL: https://github.com/apache/beam/pull/12318#issuecomment-668016046


   @aaltay @tvalentyn thank you for reviews.
   I thought it over and decided to divide it into two PRs, separate for python and for java.
   Also keeping these workflows in separate files seems to have more sense (better customization for specific language e.g. different triggering paths,  + it will not block building python wheels).
   
   Here is PR for python, I am still working on Java:
   https://github.com/apache/beam/pull/12452


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on a change in pull request #12318: [BEAM-8665] Add test suites to run Beam tests on Linux/Windows/Mac platforms.

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12318:
URL: https://github.com/apache/beam/pull/12318#discussion_r504333910



##########
File path: sdks/python/apache_beam/runners/portability/stager_test.py
##########
@@ -167,6 +167,9 @@ def test_no_main_session(self):
 
   # xdist adds unpicklable modules to the main session.
   @pytest.mark.no_xdist
+  @pytest.mark.skip(
+      sys.platform == 'win32',
+      "Windows raise TypeError: can't pickle zipimport.zipimporter objects")

Review comment:
       Put up a PR to add it back in https://github.com/apache/beam/pull/12827




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org