You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by yh...@apache.org on 2023/08/03 01:18:17 UTC
[beam] branch master updated: Add flags to build multiarch Java SDK containers and test (#27760)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 553ac1629d9 Add flags to build multiarch Java SDK containers and test (#27760)
553ac1629d9 is described below
commit 553ac1629d9e73abdc264ec54f5e0fb6c484b356
Author: Celeste Zeng <61...@users.noreply.github.com>
AuthorDate: Wed Aug 2 18:18:10 2023 -0700
Add flags to build multiarch Java SDK containers and test (#27760)
* Add a new gradle file.
* Removed all legacy worker related tasks in build.gradle, adjust tasks for multiarch containers
* Created new files for Java Dataflow ARM tests, still need to figure out Java version and Java home issues.
* Remove unused code in build.gradle.
* remove unused target, add build multiarch Java container flags, adjust build.gradle file.
* Add the TAG as env variable shared by diff Java version tests.
* Adjust dependencies.
* Change READNE for Java versions and yml file for auth.
* Made changes based on PR comments.
* Made changes based on comments, standardize yml file.
* Change github_job.
* Update .github/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml
Co-authored-by: Danny McCormick <da...@google.com>
* Update .github/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml
Co-authored-by: Danny McCormick <da...@google.com>
* remove redundant underscore
---------
Co-authored-by: Danny McCormick <da...@google.com>
---
.github/workflows/README.md | 3 +-
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 143 +++++++++++++++++++
.../google-cloud-dataflow-java/arm/build.gradle | 155 +++++++++++++++++++++
sdks/java/container/common.gradle | 7 +-
4 files changed, 306 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index 9b983d300ec..5c283e0ed81 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -117,6 +117,7 @@ Please note that jobs with matrix need to have matrix element in the comment. Ex
| Workflow name | Matrix | Trigger Phrase | Cron Status |
|:-------------:|:------:|:--------------:|:-----------:|
| [ PostCommit Go Dataflow ARM](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Go_Dataflow_ARM.yml) | N/A |`Run Go PostCommit Dataflow ARM`| [](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Go_Dataflow_ARM.yml) |
+| [ PostCommit Java Examples Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml) | N/A |`Run Java_Examples_Dataflow_ARM PostCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml) |
| [ PreCommit Community Metrics ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_CommunityMetrics.yml) | N/A |`Run CommunityMetrics PreCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_CommunityMetrics.yml) |
| [ PreCommit Go ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml) | N/A |`Run Go PreCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Go.yml) |
| [ PreCommit Java Examples Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow.yml) | N/A |`Run Java_Examples_Dataflow PreCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow.yml) |
@@ -137,4 +138,4 @@ Please note that jobs with matrix need to have matrix element in the comment. Ex
| [ PreCommit Website ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website.yml) | N/A |`Run Website PreCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website.yml) |
| [ PreCommit Website Stage GCS ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml) | N/A |`Run Website_Stage_GCS PreCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Website_Stage_GCS.yml) |
| [ PreCommit Whitespace ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Whitespace.yml) | N/A |`Run Whitespace PreCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Whitespace.yml) |
-| [ Python Validates Container Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) | ['3.8','3.9','3.10','3.11'] | `Run Python ValidatesContainer Dataflow ARM (matrix_element)`| [](https://github.com/apache/beam/actions/workflows/beam_Pytho [...]
\ No newline at end of file
+| [ Python Validates Container Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) | ['3.8','3.9','3.10','3.11'] | `Run Python ValidatesContainer Dataflow ARM (matrix_element)`| [](https://github.com/apache/beam/actions/workflows/beam_Pytho [...]
diff --git a/.github/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml b/.github/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml
new file mode 100644
index 00000000000..d886d2b3d93
--- /dev/null
+++ b/.github/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml
@@ -0,0 +1,143 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: PostCommit Java Examples Dataflow ARM
+
+on:
+ push:
+ tags: ['v*']
+ branches: ['master', 'release-*']
+ paths:
+ - 'model/**'
+ - 'sdks/java/**'
+ - 'runners/google-cloud-dataflow-java/**'
+ - 'examples/java/**'
+ - 'examples/kotlin/**'
+ - 'release/**'
+ - '.github/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml'
+ issue_comment:
+ types: [created]
+ schedule:
+ - cron: '0 */6 * * *'
+ workflow_dispatch:
+
+# This allows a subsequently queued workflow run to interrupt previous runs
+concurrency:
+ group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.sha || github.head_ref || github.ref }}-${{ github.event.sender.login }}-${{ github.event.schedule }}'
+ cancel-in-progress: true
+
+#Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
+permissions:
+ actions: write
+ pull-requests: read
+ checks: read
+ contents: read
+ deployments: read
+ id-token: none
+ issues: read
+ discussions: read
+ packages: read
+ pages: read
+ repository-projects: read
+ security-events: read
+ statuses: read
+
+env:
+ TAG: $(date +'%Y%m%d-%H%M%S%N')
+
+jobs:
+ beam_PostCommit_Java_Examples__Dataflow_ARM:
+ name: beam_PostCommit_Java_Examples__Dataflow_ARM (Run Java_Examples_Dataflow_ARM PostCommit)
+ runs-on: [self-hosted, ubuntu-20.04, main]
+ strategy:
+ fail-fast: false
+ if: |
+ github.event_name == 'push' ||
+ github.event_name == 'schedule' ||
+ github.event.comment.body == 'Run Java_Examples_Dataflow_ARM PostCommit' ||
+ github. event_name == 'workflow_dispatch'
+ steps:
+ - name: Check out repository code
+ uses: actions/checkout@v3
+ with:
+ ref: ${{ github.event.pull_request.head.sha }}
+ - name: Rerun on comment
+ if: github.event.comment.body == 'Run Java_Examples_Dataflow_ARM PostCommit'
+ uses: ./.github/actions/rerun-job-action
+ with:
+ pull_request_url: ${{ github.event.issue.pull_request.url }}
+ github_repository: ${{ github.repository }}
+ github_token: ${{ secrets.GITHUB_TOKEN }}
+ github_job: "beam_PostCommit_Java_Examples__Dataflow_ARM (Run Java_Examples_Dataflow_ARM PostCommit)"
+ github_current_run_id: ${{ github.run_id }}
+ - name: Setup self-hosted
+ uses: ./.github/actions/setup-self-hosted-action
+ with:
+ requires-py-38: false
+ requires-py-39: false
+ requires-go: false
+ - name: Setup Gradle
+ uses: gradle/gradle-build-action@v2
+ with:
+ cache-read-only: false
+ - name: Set up Docker Buildx
+ uses: docker/setup-buildx-action@v2
+ - name: Authenticate on GCP
+ id: auth
+ uses: google-github-actions/setup-gcloud@v0
+ with:
+ service_account_email: ${{ secrets.GCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.GCP_SA_KEY }}
+ project_id: ${{ secrets.GCP_PROJECT_ID }}
+ export_default_credentials: true
+ - name: GCloud Docker credential helper
+ run: |
+ gcloud auth configure-docker us.gcr.io
+ - name: run javaExamplesDataflowARMPostCommit script for Java8
+ uses: ./.github/actions/gradle-command-self-hosted-action
+ with:
+ gradle-command: :runners:google-cloud-dataflow-java:arm:examplesJavaRunnerV2IntegrationTestARM
+ arguments: |
+ -Pcontainer-architecture-list=arm64,amd64 \
+ -Ppush-containers \
+ -Pdocker-repository-root=us.gcr.io/apache-beam-testing/java-postcommit-it/arm \
+ -Pdocker-tag=${{env.TAG}} \
+ - name: run javaExamplesDataflowARMPostCommit script for Java11
+ uses: ./.github/actions/gradle-command-self-hosted-action
+ with:
+ gradle-command: :runners:google-cloud-dataflow-java:arm:examplesJavaRunnerV2IntegrationTestARM
+ arguments: |
+ -PcompileAndRunTestsWithJava11 \
+ -Pcontainer-architecture-list=arm64,amd64 \
+ -Ppush-containers \
+ -Pdocker-repository-root=us.gcr.io/apache-beam-testing/java-postcommit-it/arm \
+ -Pdocker-tag=${{env.TAG}} \
+ - name: run javaExamplesDataflowARMPostCommit script for Java17
+ uses: ./.github/actions/gradle-command-self-hosted-action
+ with:
+ gradle-command: :runners:google-cloud-dataflow-java:arm:examplesJavaRunnerV2IntegrationTestARM
+ arguments: |
+ -PcompileAndRunTestsWithJava17 \
+ -Pcontainer-architecture-list=arm64,amd64 \
+ -Ppush-containers \
+ -Pdocker-repository-root=us.gcr.io/apache-beam-testing/java-postcommit-it/arm \
+ -Pdocker-tag=${{env.TAG}} \
+ - name: Upload test report
+ uses: actions/upload-artifact@v3
+ with:
+ name: java-code-coverage-report
+ path: "**/build/test-results/**/*.xml"
diff --git a/runners/google-cloud-dataflow-java/arm/build.gradle b/runners/google-cloud-dataflow-java/arm/build.gradle
new file mode 100644
index 00000000000..8e11f1ba099
--- /dev/null
+++ b/runners/google-cloud-dataflow-java/arm/build.gradle
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import groovy.json.JsonOutput
+
+plugins { id 'org.apache.beam.module' }
+applyJavaNature(
+ automaticModuleName: 'org.apache.beam.runners.dataflow',
+ classesTriggerCheckerBugs: [
+ 'PrimitiveParDoSingleFactory': 'https://github.com/typetools/checker-framework/issues/3791',
+ // TODO(https://github.com/apache/beam/issues/21068): This currently crashes with checkerframework 3.10.0
+ // when compiling :runners:google-cloud-dataflow-java:compileJava with:
+ // message: class file for com.google.api.services.bigquery.model.TableRow not found
+ // ; The Checker Framework crashed. Please report the crash.
+ // Compilation unit: /usr/local/google/home/lcwik/git/beam/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DefaultCoderCloudObjectTranslatorRegistrar.java
+ // Last visited tree at line 57 column 1:
+ // @AutoService(CoderCloudObjectTranslatorRegistrar.class)
+ // Exception: com.sun.tools.javac.code.Symbol$CompletionFailure: class file for com.google.api.services.bigquery.model.TableRow not found; com.sun.tools.javac.code.Symbol$CompletionFailure: class file for com.google.api.services.bigquery.model.TableRow not found
+ 'DefaultCoderCloudObjectTranslatorRegistrar': 'TODO(https://github.com/apache/beam/issues/21068): Report the crash if still occurring on newest version',
+ ],
+)
+
+description = "Apache Beam :: Runners :: Google Cloud Dataflow"
+
+/*
+ * We need to rely on manually specifying these evaluationDependsOn to ensure that
+ * the following projects are evaluated before we evaluate this project. This is because
+ * we are attempting to reference parameters such as "sourceSets.test.output" directly.
+ */
+evaluationDependsOn(":sdks:java:io:google-cloud-platform")
+evaluationDependsOn(":sdks:java:core")
+evaluationDependsOn(":examples:java")
+evaluationDependsOn(":runners:google-cloud-dataflow-java:worker")
+evaluationDependsOn(":sdks:java:container:java8")
+evaluationDependsOn(":sdks:java:container:java11")
+
+processResources {
+ filter org.apache.tools.ant.filters.ReplaceTokens, tokens: [
+ 'dataflow.fnapi_environment_major_version' : '8',
+ 'dataflow.fnapi_container_version' : 'beam-master-20230426',
+ 'dataflow.container_base_repository' : 'gcr.io/cloud-dataflow/v1beta3',
+ ]
+}
+
+// Exclude tests that need a runner
+test {
+ systemProperty "beamTestPipelineOptions", ""
+ systemProperty "beamUseDummyRunner", "true"
+ useJUnit {
+ excludeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
+ }
+}
+
+dependencies {
+ dockerDependency project(path: ":runners:google-cloud-dataflow-java", configuration: "examplesJavaIntegrationTest")
+}
+
+def dataflowProject = project.findProperty('dataflowProject') ?: 'apache-beam-testing'
+def dataflowRegion = project.findProperty('dataflowRegion') ?: 'us-central1'
+def dataflowValidatesTempRoot = project.findProperty('dataflowTempRoot') ?: 'gs://temp-storage-for-validates-runner-tests'
+def firestoreDb = project.findProperty('firestoreDb') ?: 'firestoredb'
+def dockerImageRoot = project.findProperty('docker-repository-root') ?: "us.gcr.io/${dataflowProject}/java-postcommit-it"
+def DockerJavaMultiarchImageContainer = "${dockerImageRoot}/${project.docker_image_default_repo_prefix}java${javaVer}_sdk"
+def dockerTag = project.findProperty('docker-tag') ?: new Date().format('yyyyMMddHHmmss')
+ext.DockerJavaMultiarchImageName = "${DockerJavaMultiarchImageContainer}:${dockerTag}"
+
+def runnerV2PipelineOptionsARMARM = [
+ "--runner=TestDataflowRunner",
+ "--project=${dataflowProject}",
+ "--region=${dataflowRegion}",
+ "--tempRoot=${dataflowValidatesTempRoot}",
+ "--sdkContainerImage=${DockerJavaMultiarchImageContainer}:${dockerTag}",
+ "--experiments=use_unified_worker,use_runner_v2",
+ "--firestoreDb=${firestoreDb}",
+ "--machine_type=t2a-standard-1",
+]
+
+// Build and push multi-arch docker images to a container registry for use within tests.
+// NB: Tasks which consume docker images from the registry should depend on this
+// task directly ('dependsOn buildAndPushDockerJavaMultiarchContainer'). This ensures the correct
+// task ordering such that the registry doesn't get cleaned up prior to task completion.
+def buildAndPushDockerJavaMultiarchContainer = tasks.register("buildAndPushDockerJavaMultiarchContainer") {
+ def javaVer = "java8"
+ if(project.hasProperty('compileAndRunTestsWithJava17')) {
+ javaVer = "java17"
+ } else if(project.hasProperty('compileAndRunTestsWithJava11')) {
+ javaVer = "java11"
+ }
+ dependsOn ":sdks:java:container:${javaVer}:docker"
+}
+
+// Clean up built Java images
+def cleanUpDockerJavaMultiarchImages = tasks.register("cleanUpDockerJavaMultiarchImages") {
+ doLast {
+ exec {
+ commandLine "gcloud", "--quiet", "container", "images", "untag", "${DockerJavaMultiarchImageName}"
+ }
+ exec {
+ commandLine "./scripts/cleanup_untagged_gcr_images.sh", "${DockerJavaMultiarchImageContainer}"
+ }
+ }
+}
+
+afterEvaluate {
+ // Ensure all tasks which use published docker images run before they are cleaned up
+ tasks.each { t ->
+ if (t.dependsOn.contains(buildAndPushDockerJavaMultiarchContainer) && !t.name.equalsIgnoreCase('printrunnerV2PipelineOptionsARM')) {
+ t.finalizedBy cleanUpDockerJavaMultiarchImages
+ }
+ }
+}
+
+task printrunnerV2PipelineOptionsARM {
+ dependsOn buildAndPushDockerJavaMultiarchContainer
+
+ doLast {
+ println "To run a Dataflow job with runner V2 on ARM, add the following pipeline options to your command-line:"
+ println runnerV2PipelineOptionsARM.join(' ')
+ println "Please delete your image upon completion with the following command:"
+ println "gcloud container images delete --force-delete-tags ${DockerJavaMultiarchImageName}"
+ }
+}
+
+task examplesJavaRunnerV2IntegrationTestARM(type: Test) {
+ group = "Verification"
+ dependsOn buildAndPushDockerJavaMultiarchContainer
+
+ systemProperty "beamTestPipelineOptions", JsonOutput.toJson(runnerV2PipelineOptionsARM)
+
+ include '**/*IT.class'
+ exclude '**/TopWikipediaSessionsIT.class'
+ exclude '**/AutoCompleteIT.class'
+ // TODO(https://github.com/apache/beam/issues/20593): test times out.
+ exclude '**/FhirIOReadIT.class'
+
+ maxParallelForks 4
+ classpath = examplesJavaIntegrationTest
+ testClassesDirs = files(project(":examples:java").sourceSets.test.output.classesDirs)
+ useJUnit { }
+}
\ No newline at end of file
diff --git a/sdks/java/container/common.gradle b/sdks/java/container/common.gradle
index 5b2130f2781..bf4c122ca91 100644
--- a/sdks/java/container/common.gradle
+++ b/sdks/java/container/common.gradle
@@ -104,6 +104,9 @@ task validateJavaHome {
}
}
+def useBuildx = project.containerPlatforms() != [project.nativeArchitecture()]
+def pushContainers = project.rootProject.hasProperty(["isRelease"]) || project.rootProject.hasProperty("push-containers")
+
docker {
name containerImageName(
name: "${project.docker_image_default_repo_prefix}java${imageJavaVersion}_sdk",
@@ -121,8 +124,10 @@ docker {
project.rootProject.hasProperty(["isRelease"]),
'java_version': imageJavaVersion,
])
- buildx project.containerPlatforms() != [project.nativeArchitecture()]
+ buildx useBuildx
platform(*project.containerPlatforms())
+ load useBuildx && !pushContainers
+ push pushContainers
}
if (project.rootProject.hasProperty(["docker-pull-licenses"]) ||