You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/15 00:25:09 UTC
[GitHub] [beam] tysonjh commented on a change in pull request #12541: [BEAM-10682] Add workflow to run Java tests on Linux/Windows/Mac
tysonjh commented on a change in pull request #12541:
URL: https://github.com/apache/beam/pull/12541#discussion_r470903163
##########
File path: .github/workflows/cancel.yml
##########
@@ -32,3 +35,18 @@ jobs:
with:
workflow_id: ${{ env.WORKFLOW_ID }}
access_token: ${{ github.token }}
+
+ cancel_java_tests:
Review comment:
Is there an equivalent for python or go tests somewhere?
##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+ workflow_dispatch:
+ inputs:
+ runDataflow:
+ description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+ default: false
+ schedule:
+ - cron: '10 2 * * *'
+ push:
+ branches: ['master', 'release-*']
+ tags: 'v*'
+ pull_request:
+ branches: ['master', 'release-*']
+ tags: 'v*'
+ paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+ check_gcp_variables:
+ timeout-minutes: 5
+ name: "Check GCP variables set"
+ runs-on: ubuntu-latest
+ outputs:
+ gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+ steps:
+ - uses: actions/checkout@v2
+ - name: "Check are GCP variables set"
+ run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+ id: check_gcp_variables
+ env:
+ GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+ GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+ GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+ GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+ java_unit_tests:
+ name: 'Java Unit Tests'
+ runs-on: ${{ matrix.os }}
+ strategy:
+ fail-fast: false
+ matrix:
+ os: [ubuntu-latest, macos-latest, windows-latest]
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ # :sdks:java:core:test
+ - name: Run :sdks:java:core:test
+ run: ./gradlew :sdks:java:core:test
+ - name: Upload test logs for :sdks:java:core:test
+ uses: actions/upload-artifact@v2
+ if: always()
Review comment:
If `if: always()` is always true, is this condition really needed? (here and elsewhere in this workflow)
##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+ workflow_dispatch:
+ inputs:
+ runDataflow:
+ description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+ default: false
+ schedule:
+ - cron: '10 2 * * *'
+ push:
+ branches: ['master', 'release-*']
+ tags: 'v*'
+ pull_request:
+ branches: ['master', 'release-*']
+ tags: 'v*'
+ paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+ check_gcp_variables:
+ timeout-minutes: 5
+ name: "Check GCP variables set"
+ runs-on: ubuntu-latest
+ outputs:
+ gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+ steps:
+ - uses: actions/checkout@v2
+ - name: "Check are GCP variables set"
+ run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+ id: check_gcp_variables
+ env:
+ GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+ GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+ GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+ GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+ java_unit_tests:
+ name: 'Java Unit Tests'
+ runs-on: ${{ matrix.os }}
+ strategy:
+ fail-fast: false
+ matrix:
+ os: [ubuntu-latest, macos-latest, windows-latest]
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ # :sdks:java:core:test
+ - name: Run :sdks:java:core:test
+ run: ./gradlew :sdks:java:core:test
+ - name: Upload test logs for :sdks:java:core:test
+ uses: actions/upload-artifact@v2
+ if: always()
+ with:
+ name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}
+ path: sdks/java/core/build/reports/tests/test/**
+ # :sdks:java:harness:test
+ - name: Run :sdks:java:harness:test
+ if: ${{ always() }}
Review comment:
```suggestion
if: always()
```
##########
File path: .github/workflows/cancel.yml
##########
@@ -32,3 +35,18 @@ jobs:
with:
workflow_id: ${{ env.WORKFLOW_ID }}
access_token: ${{ github.token }}
+
+ cancel_java_tests:
+ name: 'Cancel Previous Runs of Java Tests'
+ runs-on: ubuntu-latest
+ timeout-minutes: 3
+ steps:
+ - name: Get build_wheels workflow id
Review comment:
```suggestion
- name: Get java_tests workflow id
```
##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+ workflow_dispatch:
+ inputs:
+ runDataflow:
+ description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+ default: false
+ schedule:
+ - cron: '10 2 * * *'
+ push:
+ branches: ['master', 'release-*']
+ tags: 'v*'
+ pull_request:
+ branches: ['master', 'release-*']
+ tags: 'v*'
+ paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+ check_gcp_variables:
+ timeout-minutes: 5
+ name: "Check GCP variables set"
+ runs-on: ubuntu-latest
+ outputs:
+ gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+ steps:
+ - uses: actions/checkout@v2
+ - name: "Check are GCP variables set"
+ run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+ id: check_gcp_variables
+ env:
+ GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+ GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+ GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+ GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+ java_unit_tests:
+ name: 'Java Unit Tests'
+ runs-on: ${{ matrix.os }}
+ strategy:
+ fail-fast: false
+ matrix:
+ os: [ubuntu-latest, macos-latest, windows-latest]
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ # :sdks:java:core:test
+ - name: Run :sdks:java:core:test
+ run: ./gradlew :sdks:java:core:test
+ - name: Upload test logs for :sdks:java:core:test
+ uses: actions/upload-artifact@v2
+ if: always()
+ with:
+ name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}
Review comment:
Is there a risk that concurrently running workflows could race to write this artifact since it doesn't contain any globally unique information (e.g. date)? Or are artifact names unique to each individual workflow run?
##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+ workflow_dispatch:
+ inputs:
+ runDataflow:
+ description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+ default: false
+ schedule:
+ - cron: '10 2 * * *'
+ push:
+ branches: ['master', 'release-*']
+ tags: 'v*'
+ pull_request:
+ branches: ['master', 'release-*']
+ tags: 'v*'
+ paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+ check_gcp_variables:
+ timeout-minutes: 5
+ name: "Check GCP variables set"
+ runs-on: ubuntu-latest
+ outputs:
+ gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+ steps:
+ - uses: actions/checkout@v2
+ - name: "Check are GCP variables set"
+ run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+ id: check_gcp_variables
+ env:
+ GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+ GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+ GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+ GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+ java_unit_tests:
+ name: 'Java Unit Tests'
+ runs-on: ${{ matrix.os }}
+ strategy:
+ fail-fast: false
+ matrix:
+ os: [ubuntu-latest, macos-latest, windows-latest]
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ # :sdks:java:core:test
+ - name: Run :sdks:java:core:test
+ run: ./gradlew :sdks:java:core:test
+ - name: Upload test logs for :sdks:java:core:test
+ uses: actions/upload-artifact@v2
+ if: always()
+ with:
+ name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}
+ path: sdks/java/core/build/reports/tests/test/**
+ # :sdks:java:harness:test
+ - name: Run :sdks:java:harness:test
+ if: ${{ always() }}
+ run: ./gradlew :sdks:java:harness:test
+ - name: Upload test logs for :sdks:java:harness:test
+ uses: actions/upload-artifact@v2
+ if: always()
+ with:
+ name: java_unit_tests-sdks-java-harness-test-${{ matrix.os }}
+ path: sdks/java/harness/build/reports/tests/test/**
+ # :runners:core-java:test
+ - name: Run :runners:core-java:test
+ if: ${{ always() }}
+ run: ./gradlew :runners:core-java:test
+ - name: Upload test logs for :runners:core-java:test
+ uses: actions/upload-artifact@v2
+ if: always()
+ with:
+ name: java_unit_tests-runners-core-java-test-${{ matrix.os }}
+ path: runners/core-java/build/reports/tests/test/**
+
+ java_wordcount_direct_runner:
+ name: 'Java Wordcount Direct Runner'
+ runs-on: ${{ matrix.os }}
+ strategy:
+ fail-fast: false
+ matrix:
+ os: [ubuntu-latest, macos-latest, windows-latest]
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Run WordCount Unix
+ shell: bash
+ run: |
+ bash ./gradlew -p examples/ \
+ integrationTest -DintegrationTestPipelineOptions='[
+ "--runner=DirectRunner", "--tempRoot=./tmp" ]' \
+ --tests org.apache.beam.examples.WordCountIT \
+ -DintegrationTestRunner=direct
+ - name: Upload test logs
+ uses: actions/upload-artifact@v2
+ if: always()
+ with:
+ name: java_wordcount_direct_runner-${{matrix.os}}
+ path: examples/java/build/reports/tests/integrationTest
Review comment:
Why upload the whole directory here instead of a globstar? Just curious.
##########
File path: .github/workflows/java_tests.yml
##########
@@ -0,0 +1,160 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# To learn more about GitHub Actions in Apache Beam check the CI.md
+
+name: Java Tests
+
+on:
+ workflow_dispatch:
+ inputs:
+ runDataflow:
+ description: 'Type "true" if you want to run Dataflow tests (GCP variables must be configured, check CI.md)'
+ default: false
+ schedule:
+ - cron: '10 2 * * *'
+ push:
+ branches: ['master', 'release-*']
+ tags: 'v*'
+ pull_request:
+ branches: ['master', 'release-*']
+ tags: 'v*'
+ paths: ['sdks/java/**', 'model/**', 'runners/**', 'examples/java/**', 'examples/kotlin/**', 'release/**']
+
+
+jobs:
+
+ check_gcp_variables:
+ timeout-minutes: 5
+ name: "Check GCP variables set"
+ runs-on: ubuntu-latest
+ outputs:
+ gcp-variables-set: ${{ steps.check_gcp_variables.outputs.gcp-variables-set }}
+ steps:
+ - uses: actions/checkout@v2
+ - name: "Check are GCP variables set"
+ run: "./scripts/ci/ci_check_are_gcp_variables_set.sh"
+ id: check_gcp_variables
+ env:
+ GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
+ GCP_SA_EMAIL: ${{ secrets.GCP_SA_EMAIL }}
+ GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
+ GCP_TESTING_BUCKET: ${{ secrets.GCP_TESTING_BUCKET }}
+
+ java_unit_tests:
+ name: 'Java Unit Tests'
+ runs-on: ${{ matrix.os }}
+ strategy:
+ fail-fast: false
+ matrix:
+ os: [ubuntu-latest, macos-latest, windows-latest]
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ # :sdks:java:core:test
+ - name: Run :sdks:java:core:test
+ run: ./gradlew :sdks:java:core:test
+ - name: Upload test logs for :sdks:java:core:test
+ uses: actions/upload-artifact@v2
+ if: always()
+ with:
+ name: java_unit_tests-sdks-java-core-test-${{ matrix.os }}
+ path: sdks/java/core/build/reports/tests/test/**
+ # :sdks:java:harness:test
+ - name: Run :sdks:java:harness:test
+ if: ${{ always() }}
+ run: ./gradlew :sdks:java:harness:test
+ - name: Upload test logs for :sdks:java:harness:test
+ uses: actions/upload-artifact@v2
+ if: always()
+ with:
+ name: java_unit_tests-sdks-java-harness-test-${{ matrix.os }}
+ path: sdks/java/harness/build/reports/tests/test/**
+ # :runners:core-java:test
+ - name: Run :runners:core-java:test
+ if: ${{ always() }}
+ run: ./gradlew :runners:core-java:test
+ - name: Upload test logs for :runners:core-java:test
+ uses: actions/upload-artifact@v2
+ if: always()
+ with:
+ name: java_unit_tests-runners-core-java-test-${{ matrix.os }}
+ path: runners/core-java/build/reports/tests/test/**
+
+ java_wordcount_direct_runner:
+ name: 'Java Wordcount Direct Runner'
+ runs-on: ${{ matrix.os }}
+ strategy:
+ fail-fast: false
+ matrix:
+ os: [ubuntu-latest, macos-latest, windows-latest]
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Run WordCount Unix
+ shell: bash
+ run: |
Review comment:
Why is a shell used in some jobs but not others (specified here as a config and used in the run config below)? Just curious if it is needed for some reason here specifically since it is an integration test.
##########
File path: CI.md
##########
@@ -75,8 +75,27 @@ run categories. Here is a summary of the run categories with regards of the jobs
Those jobs often have matrix run strategy which runs several different variations of the jobs
(with different platform type / Python version to run for example)
+### Google Cloud Platform Credentials
+
+Some of the jobs require variables stored as a [GitHub Secrets](https://docs.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets)
+to perform operations on Google Cloud Platform. Currently these jobs are limited to Apache repository only.
Review comment:
A user can set these secrets themselves though to test on their own fork?
##########
File path: sdks/java/core/src/test/java/org/apache/beam/sdk/io/DefaultFilenamePolicyTest.java
##########
@@ -28,6 +31,12 @@
@RunWith(JUnit4.class)
public class DefaultFilenamePolicyTest {
+ @Before
+ public void setup() {
+ // TODO: Fix failing on windows, https://issues.apache.org/jira/browse/BEAM-10683.
Review comment:
Can you create a Jira issue per failing test suite as subtasks of BEAM-10683 please? Having a list of failing test suites will be easier for tracking and hopefully encourage people to chip away at it instead of one large issue for them all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org