You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by yh...@apache.org on 2025/06/05 14:54:47 UTC

(beam) branch master updated: Add postcommit yaml xlang workflow and split tests accordingly (#35119)

This is an automated email from the ASF dual-hosted git repository.

yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
     new e6f50a84501 Add postcommit yaml xlang workflow and split tests accordingly  (#35119)
e6f50a84501 is described below

commit e6f50a84501cc15c11f50c04bc7f1f5e744d8387
Author: Derrick Williams <de...@google.com>
AuthorDate: Thu Jun 5 10:54:39 2025 -0400

    Add postcommit yaml xlang workflow and split tests accordingly  (#35119)
    
    * move precommit yaml xlang to postcommit
    
    * add json file to trigger path
    
    * update pull request targets
    
    * add readme workflow changes
    
    * add cloud sdk setup
    
    * switch out to ubuntu-latest
    
    * add back precommit workflow
    
    * switch names and add postcommit
    
    * switch out to postCommitYamlIT
    
    * add test_files_dir flag
    
    * add conftest.py file for capturing directory flag
    
    * shift yaml tests around to appropriate locations
    
    * add back precommit to readme
    
    * add license for conftest.py
    
    * revert precommit to previous name
    
    * remove github.event.comment.body trigger
---
 .../beam_PostCommit_Yaml_Xlang_Direct.json         |  4 +++
 .github/workflows/README.md                        |  1 +
 ...t.yml => beam_PostCommit_Yaml_Xlang_Direct.yml} | 40 ++++++++--------------
 .../workflows/beam_PreCommit_Yaml_Xlang_Direct.yml |  2 +-
 sdks/python/apache_beam/yaml/conftest.py           | 34 ++++++++++++++++++
 .../yaml/{tests => extended_tests}/bigquery.yaml   |  0
 .../yaml/{tests => extended_tests}/enrichment.yaml |  0
 .../yaml/{tests => extended_tests}/spanner.yaml    |  0
 sdks/python/apache_beam/yaml/integration_tests.py  | 10 +++++-
 .../assign_timestamps.yaml                         |  0
 .../yaml/{extended_tests => tests}/create.yaml     |  0
 .../extract_windowing_Info.yaml                    |  0
 .../yaml/{extended_tests => tests}/filter.yaml     |  0
 .../yaml/{extended_tests => tests}/flatten.yaml    |  0
 .../{extended_tests => tests}/ml_transform.yaml    |  0
 .../yaml/{extended_tests => tests}/partition.yaml  |  0
 .../{extended_tests => tests}/pytransform.yaml     |  0
 .../{extended_tests => tests}/runinference.yaml    |  0
 .../validate_with_schema.yaml                      |  0
 .../yaml/{extended_tests => tests}/windowinto.yaml |  0
 sdks/python/build.gradle                           | 21 +++++++++++-
 21 files changed, 83 insertions(+), 29 deletions(-)

diff --git a/.github/trigger_files/beam_PostCommit_Yaml_Xlang_Direct.json b/.github/trigger_files/beam_PostCommit_Yaml_Xlang_Direct.json
new file mode 100644
index 00000000000..a975cd1cd10
--- /dev/null
+++ b/.github/trigger_files/beam_PostCommit_Yaml_Xlang_Direct.json
@@ -0,0 +1,4 @@
+{
+  "comment": "Modify this file in a trivial way to cause this test suite to run",
+  "revision": 1
+}
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index b9069c530e5..a7cdf99d53c 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -401,6 +401,7 @@ PostCommit Jobs run in a schedule against master branch and generally do not get
 | [ PostCommit XVR PythonUsingJavaSQL Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml) | N/A |`beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.json`| [![.github/workflows/beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_PythonU [...]
 | [ PostCommit XVR Samza ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_Samza.yml) | N/A |`beam_PostCommit_XVR_Samza.json`| [![.github/workflows/beam_PostCommit_XVR_Samza.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_Samza.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_Samza.yml?query=event%3Aschedule) |
 | [ PostCommit XVR Spark3 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_Spark3.yml) | N/A |`beam_PostCommit_XVR_Spark3.json`| [![.github/workflows/beam_PostCommit_XVR_Spark3.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_Spark3.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_Spark3.yml?query=event%3Aschedule) |
+| [ PostCommit YAML Xlang Direct ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml) | N/A |`beam_PostCommit_Yaml_Xlang_Direct.json`| [![.github/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml?query=event%3Aschedule) |
 | [ Python Validates Container Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) | ['3.8','3.9','3.10','3.11'] |`beam_Python_ValidatesContainer_Dataflow_ARM.json`|[![.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesCo [...]
 
 ### PerformanceTests and Benchmark Jobs
diff --git a/.github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml b/.github/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml
similarity index 75%
copy from .github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml
copy to .github/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml
index 92a20e5003e..93efb4f4f0c 100644
--- a/.github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml
+++ b/.github/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml
@@ -13,27 +13,13 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-name: PreCommit YAML Xlang Direct
+name: PostCommit YAML Xlang Direct
 
 on:
-  pull_request_target:
-    paths: ['release/trigger_all_tests.json', 'model/**', 'sdks/python/**']
-  issue_comment:
-    types: [created]
-  push:
-    tags: ['v*']
-    branches: ['master', 'release-*']
-    paths:
-      - "model/**"
-      - "release/**"
-      - "sdks/python/**"
-      - "sdks/java/extensions/schemaio-expansion-service/**"
-      - "sdks/java/extensions/sql/**"
-      - "sdks/java/io/expansion-service/**"
-      - "sdks/java/io/google-cloud-platform/**"
-      - ".github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml"
   schedule:
     - cron: '30 5/6 * * *'
+  pull_request_target:
+    paths: ['release/trigger_all_tests.json', '.github/trigger_files/beam_PostCommit_Yaml_Xlang_Direct.json']
   workflow_dispatch:
 
 #Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
@@ -63,19 +49,18 @@ env:
   GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
 
 jobs:
-  beam_PreCommit_Yaml_Xlang_Direct:
+  beam_PostCommit_Yaml_Xlang_Direct:
     if: |
       github.event_name == 'workflow_dispatch' ||
       github.event_name == 'pull_request_target' ||
-      (github.event_name == 'schedule' && github.repository == 'apache/beam') ||
-      github.event.comment.body == 'Run Yaml_Xlang_Direct PreCommit'
-    runs-on: [self-hosted, ubuntu-20.04, main]
+      (github.event_name == 'schedule' && github.repository == 'apache/beam')
+    runs-on: [self-hosted, ubuntu-latest, main]
     timeout-minutes: 100
     name: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
     strategy:
       matrix:
-        job_name: ["beam_PreCommit_Yaml_Xlang_Direct"]
-        job_phrase: ["Run Yaml_Xlang_Direct PreCommit"]
+        job_name: ["beam_PostCommit_Yaml_Xlang_Direct"]
+        job_phrase: ["Run Yaml_Xlang_Direct PostCommit"]
     steps:
       - uses: actions/checkout@v4
       - name: Setup repository
@@ -88,10 +73,13 @@ jobs:
         uses: ./.github/actions/setup-environment-action
         with:
           python-version: default
-      - name: run PreCommit Yaml Xlang Direct script
+          java-version: '11'
+      - name: Set up Cloud SDK
+        uses: google-github-actions/setup-gcloud@v2
+      - name: run PostCommit Yaml Xlang Direct script
         uses: ./.github/actions/gradle-command-self-hosted-action
         with:
-          gradle-command: :sdks:python:yamlIntegrationTests -PbeamPythonExtra=ml_test
+          gradle-command: :sdks:python:postCommitYamlIntegrationTests -PbeamPythonExtra=ml_test
       - name: Archive Python Test Results
         uses: actions/upload-artifact@v4
         if: failure()
@@ -105,4 +93,4 @@ jobs:
           commit: '${{ env.prsha || env.GITHUB_SHA }}'
           comment_mode: ${{ github.event_name == 'issue_comment'  && 'always' || 'off' }}
           files: '**/pytest*.xml'
-          large_files: true
+          large_files: true
\ No newline at end of file
diff --git a/.github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml b/.github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml
index 92a20e5003e..2e6a33f6688 100644
--- a/.github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml
+++ b/.github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml
@@ -105,4 +105,4 @@ jobs:
           commit: '${{ env.prsha || env.GITHUB_SHA }}'
           comment_mode: ${{ github.event_name == 'issue_comment'  && 'always' || 'off' }}
           files: '**/pytest*.xml'
-          large_files: true
+          large_files: true
\ No newline at end of file
diff --git a/sdks/python/apache_beam/yaml/conftest.py b/sdks/python/apache_beam/yaml/conftest.py
new file mode 100644
index 00000000000..e8fb2c61581
--- /dev/null
+++ b/sdks/python/apache_beam/yaml/conftest.py
@@ -0,0 +1,34 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This global will be set by pytest_configure
+# and can be imported by other modules.
+# Initialize with a default that matches the default.
+yaml_test_files_dir = "tests"
+
+
+def pytest_addoption(parser):
+  parser.addoption(
+      "--test_files_dir",
+      action="store",
+      default="tests",
+      help="Directory with YAML test files, relative to integration_tests.py")
+
+
+def pytest_configure(config):
+  global yaml_test_files_dir
+  yaml_test_files_dir = config.getoption("test_files_dir")
diff --git a/sdks/python/apache_beam/yaml/tests/bigquery.yaml b/sdks/python/apache_beam/yaml/extended_tests/bigquery.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/tests/bigquery.yaml
rename to sdks/python/apache_beam/yaml/extended_tests/bigquery.yaml
diff --git a/sdks/python/apache_beam/yaml/tests/enrichment.yaml b/sdks/python/apache_beam/yaml/extended_tests/enrichment.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/tests/enrichment.yaml
rename to sdks/python/apache_beam/yaml/extended_tests/enrichment.yaml
diff --git a/sdks/python/apache_beam/yaml/tests/spanner.yaml b/sdks/python/apache_beam/yaml/extended_tests/spanner.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/tests/spanner.yaml
rename to sdks/python/apache_beam/yaml/extended_tests/spanner.yaml
diff --git a/sdks/python/apache_beam/yaml/integration_tests.py b/sdks/python/apache_beam/yaml/integration_tests.py
index d8c04d9718f..f6c60ae5f12 100644
--- a/sdks/python/apache_beam/yaml/integration_tests.py
+++ b/sdks/python/apache_beam/yaml/integration_tests.py
@@ -53,6 +53,7 @@ from apache_beam.options.pipeline_options import PipelineOptions
 from apache_beam.utils import python_callable
 from apache_beam.yaml import yaml_provider
 from apache_beam.yaml import yaml_transform
+from apache_beam.yaml.conftest import yaml_test_files_dir
 
 
 @contextlib.contextmanager
@@ -603,8 +604,15 @@ def parse_test_files(filepattern):
       globals()[suite_name] = type(suite_name, (unittest.TestCase, ), methods)
 
 
+# Logging setup
 logging.getLogger().setLevel(logging.INFO)
-parse_test_files(os.path.join(os.path.dirname(__file__), 'tests', '*.yaml'))
+
+# Dynamically create test methods from the tests directory.
+# yaml_test_files_dir comes from conftest.py and set by pytest_configure.
+_test_files_dir = yaml_test_files_dir
+_file_pattern = os.path.join(
+    os.path.dirname(__file__), _test_files_dir, '*.yaml')
+parse_test_files(_file_pattern)
 
 if __name__ == '__main__':
   logging.getLogger().setLevel(logging.INFO)
diff --git a/sdks/python/apache_beam/yaml/extended_tests/assign_timestamps.yaml b/sdks/python/apache_beam/yaml/tests/assign_timestamps.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/extended_tests/assign_timestamps.yaml
rename to sdks/python/apache_beam/yaml/tests/assign_timestamps.yaml
diff --git a/sdks/python/apache_beam/yaml/extended_tests/create.yaml b/sdks/python/apache_beam/yaml/tests/create.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/extended_tests/create.yaml
rename to sdks/python/apache_beam/yaml/tests/create.yaml
diff --git a/sdks/python/apache_beam/yaml/extended_tests/extract_windowing_Info.yaml b/sdks/python/apache_beam/yaml/tests/extract_windowing_Info.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/extended_tests/extract_windowing_Info.yaml
rename to sdks/python/apache_beam/yaml/tests/extract_windowing_Info.yaml
diff --git a/sdks/python/apache_beam/yaml/extended_tests/filter.yaml b/sdks/python/apache_beam/yaml/tests/filter.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/extended_tests/filter.yaml
rename to sdks/python/apache_beam/yaml/tests/filter.yaml
diff --git a/sdks/python/apache_beam/yaml/extended_tests/flatten.yaml b/sdks/python/apache_beam/yaml/tests/flatten.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/extended_tests/flatten.yaml
rename to sdks/python/apache_beam/yaml/tests/flatten.yaml
diff --git a/sdks/python/apache_beam/yaml/extended_tests/ml_transform.yaml b/sdks/python/apache_beam/yaml/tests/ml_transform.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/extended_tests/ml_transform.yaml
rename to sdks/python/apache_beam/yaml/tests/ml_transform.yaml
diff --git a/sdks/python/apache_beam/yaml/extended_tests/partition.yaml b/sdks/python/apache_beam/yaml/tests/partition.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/extended_tests/partition.yaml
rename to sdks/python/apache_beam/yaml/tests/partition.yaml
diff --git a/sdks/python/apache_beam/yaml/extended_tests/pytransform.yaml b/sdks/python/apache_beam/yaml/tests/pytransform.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/extended_tests/pytransform.yaml
rename to sdks/python/apache_beam/yaml/tests/pytransform.yaml
diff --git a/sdks/python/apache_beam/yaml/extended_tests/runinference.yaml b/sdks/python/apache_beam/yaml/tests/runinference.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/extended_tests/runinference.yaml
rename to sdks/python/apache_beam/yaml/tests/runinference.yaml
diff --git a/sdks/python/apache_beam/yaml/extended_tests/validate_with_schema.yaml b/sdks/python/apache_beam/yaml/tests/validate_with_schema.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/extended_tests/validate_with_schema.yaml
rename to sdks/python/apache_beam/yaml/tests/validate_with_schema.yaml
diff --git a/sdks/python/apache_beam/yaml/extended_tests/windowinto.yaml b/sdks/python/apache_beam/yaml/tests/windowinto.yaml
similarity index 100%
rename from sdks/python/apache_beam/yaml/extended_tests/windowinto.yaml
rename to sdks/python/apache_beam/yaml/tests/windowinto.yaml
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index d4708803b0f..accacb1953b 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -125,7 +125,7 @@ tasks.register("generateYamlDocs") {
 }
 
 tasks.register("yamlIntegrationTests") {
-  description "Runs integration tests for yaml pipelines."
+  description "Runs precommit integration tests for yaml pipelines."
 
   dependsOn installGcpTest
   // Need to build all expansion services referenced in apache_beam/yaml/*.*
@@ -143,6 +143,25 @@ tasks.register("yamlIntegrationTests") {
   }
 }
 
+tasks.register("postCommitYamlIntegrationTests") {
+  description "Runs postcommit integration tests for yaml pipelines."
+
+  dependsOn installGcpTest
+  // Need to build all expansion services referenced in apache_beam/yaml/*.*
+  // grep -oh 'sdk.*Jar' sdks/python/apache_beam/yaml/*.yaml | sort | uniq
+  dependsOn ":sdks:java:extensions:schemaio-expansion-service:shadowJar"
+  dependsOn ":sdks:java:extensions:sql:expansion-service:shadowJar"
+  dependsOn ":sdks:java:io:expansion-service:build"
+  dependsOn ":sdks:java:io:google-cloud-platform:expansion-service:build"
+
+  doLast {
+    exec {
+      executable 'sh'
+      args '-c', "${envdir}/bin/pytest -v apache_beam/yaml/integration_tests.py --deselect apache_beam/yaml/integration_tests.py::BigqueryTest::test_ReadFromBigQuery_ExternalJavaProvider_0 --test_files_dir='extended_tests'"
+    }
+  }
+}
+
 // Create Python wheels for given platform and Python version
 // build identifiers for cibuildwheel
 def platform_identifiers_map = [