You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by yh...@apache.org on 2024/03/22 13:57:28 UTC
(beam) branch master updated: Misc doc and infra fix after 2.55.0 release (#30702)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 33a0bfb5d37 Misc doc and infra fix after 2.55.0 release (#30702)
33a0bfb5d37 is described below
commit 33a0bfb5d374fae99b1f19405aba341a0716b1c0
Author: Yi Hu <ya...@google.com>
AuthorDate: Fri Mar 22 09:57:21 2024 -0400
Misc doc and infra fix after 2.55.0 release (#30702)
* Release doc fixes
* Update workflow README for newly added workflows
* Do not trigger sickbay test in release verification
* Remove redundant shadowjar validation exclusions in harness build.gradle
* Use API token in deploy_rc_pypi action, in alignment with finalize release
* Unsickbay test_bigtable_write
---
.github/build.gradle | 2 +-
.github/workflows/README.md | 12 +++++++-----
.github/workflows/beam_PostCommit_Java_Sickbay.yml | 2 +-
.github/workflows/beam_PostCommit_Sickbay_Python.yml | 2 +-
.github/workflows/deploy_release_candidate_pypi.yaml | 13 +++++--------
contributor-docs/release-guide.md | 2 +-
sdks/java/harness/build.gradle | 15 ---------------
.../apache_beam/examples/cookbook/bigtableio_it_test.py | 4 +---
8 files changed, 17 insertions(+), 35 deletions(-)
diff --git a/.github/build.gradle b/.github/build.gradle
index de267054562..8105d813018 100644
--- a/.github/build.gradle
+++ b/.github/build.gradle
@@ -50,7 +50,7 @@ task check {
// precommit and postcommit should triggered by this specific file
// this is to ensure not missing test during release branch verification
- if (paths != null && !paths.contains('release/trigger_all_tests.json')) {
+ if (paths != null && !paths.contains('release/trigger_all_tests.json') && !fname.toLowerCase().contains('sickbay')) {
errors.add("Error validating ${fname}: " +
"Please add 'release/trigger_all_tests.json' to the trigger path")
return
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index 128279cd956..74c9ee90868 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -215,8 +215,7 @@ PreCommit Jobs run in a schedule and also get triggered in a PR if relevant sour
| [ PreCommit Java Debezium IO Direct ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Debezium_IO_Direct.yml) | N/A |`Run Java_Debezium_IO_Direct PreCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Debezium_IO_Direct.yml?query=event%3Aschedule) |
| [ PreCommit Java ElasticSearch IO Direct ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_ElasticSearch_IO_Direct.yml) | N/A |`Run Java_ElasticSearch_IO_Direct PreCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_ElasticSearch_IO_Direct.yml?query [...]
| [ PreCommit Java Examples Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow.yml) | N/A |`Run Java_Examples_Dataflow PreCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow.yml?query=event%3Aschedule) |
-| [ PreCommit Java Examples Dataflow Java11 ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow_Java11.yml) | N/A | `Run Java_Examples_Dataflow_Java11 PreCommit` | [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow_Java11.y [...]
-| [ PreCommit Java Examples Dataflow Java17 ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow_Java17.yml) | N/A | `Run Java_Examples_Dataflow_Java17 PreCommit` | [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow_Java17.y [...]
+| [ PreCommit Java Examples Dataflow Java21 ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow_Java21.yml) | N/A | `Run Java_Examples_Dataflow_Java21 PreCommit` | [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Examples_Dataflow_Java21.y [...]
| [ PreCommit Java File-schema-transform IO Direct ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_File-schema-transform_IO_Direct.yml) | N/A |`Run Java_File-schema-transform_IO_Direct PreCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PreCommi [...]
| [ PreCommit Java Flink Versions ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Flink_Versions.yml) | N/A |`Run Java_Flink_Versions PreCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_Flink_Versions.yml?query=event%3Aschedule) |
| [ PreCommit Java GCP IO Direct ](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_GCP_IO_Direct.yml) | N/A |`Run Java_GCP_IO_Direct PreCommit`| [](https://github.com/apache/beam/actions/workflows/beam_PreCommit_Java_GCP_IO_Direct.yml?query=event%3Aschedule) |
@@ -285,6 +284,7 @@ PostCommit Jobs run in a schedule against master branch and generally do not get
| [ PostCommit Go VR Samza](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Go_VR_Samza.yml) | N/A |`beam_PostCommit_Go_VR_Samza.json`| [](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Go_VR_Samza.yml?query=event%3Aschedule) |
| [ PostCommit Go VR Spark](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Go_VR_Spark.yml) | N/A |`beam_PostCommit_Go_VR_Spark.json`| [](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Go_VR_Spark.yml?query=event%3Aschedule) |
| [ PostCommit Java Avro Versions ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Avro_Versions.yml) | N/A |`beam_PostCommit_Java_Avro_Versions.json`| [](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Avro_Versions.yml?query=event%3Aschedule) |
+| [ PostCommit Java BigQueryEarlyRollout ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_BigQueryEarlyRollout.yml) | N/A |`beam_PostCommit_Java_BigQueryEarlyRollout.json`| [](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_BigQueryEarlyRollout.yml?query=event [...]
| [ PostCommit Java Dataflow V1 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_DataflowV1.yml) | N/A |`beam_PostCommit_Java_DataflowV1.json`| [](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_DataflowV1.yml?query=event%3Aschedule) |
| [ PostCommit Java Dataflow V2 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_DataflowV2.yml) | N/A |`beam_PostCommit_Java_DataflowV2.json`| [](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_DataflowV2.yml?query=event%3Aschedule) |
| [ PostCommit Java Examples Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml) | ['8','11','17','21'] |`beam_PostCommit_Java_Examples_Dataflow_ARM.json`| [](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Examples_Dataf [...]
@@ -471,15 +471,17 @@ PostCommit Jobs run in a schedule against master branch and generally do not get
| Workflow name | Matrix | Cron Status |
|:-------------:|:------:|:-----------:|
-| [ Publish BeamMetrics ](https://github.com/apache/beam/actions/workflows/beam_Publish_BeamMetrics.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_Publish_BeamMetrics.yml?query=event%3Aschedule)
-| [ Publish Website ](https://github.com/apache/beam/actions/workflows/beam_Publish_Website.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_Publish_Website.yml?query=event%3Aschedule) |
| [ Cancel Stale Dataflow Jobs ](https://github.com/apache/beam/actions/workflows/beam_CancelStaleDataflowJobs.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_CancelStaleDataflowJobs.yml?query=event%3Aschedule) |
| [ Clean Up GCP Resources ](https://github.com/apache/beam/actions/workflows/beam_CleanUpGCPResources.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_CleanUpGCPResources.yml?query=event%3Aschedule) |
| [ Clean Up Prebuilt SDK Images ](https://github.com/apache/beam/actions/workflows/beam_CleanUpPrebuiltSDKImages.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_CleanUpPrebuiltSDKImages.yml?query=event%3Aschedule) |
| [ Cleanup Dataproc Resources ](https://github.com/apache/beam/actions/workflows/beam_CleanUpDataprocResources.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_CleanUpDataprocResources.yml?query=event%3Aschedule)
| [ Community Metrics Prober ](https://github.com/apache/beam/actions/workflows/beam_Prober_CommunityMetrics.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_Prober_CommunityMetrics.yml?query=event%3Aschedule)
+| [ Playground CI Nightly ](https://github.com/apache/beam/actions/workflows/beam_Playground_CI_Nightly.yml) | ["java", "go", "python"] | [](https://github.com/apache/beam/actions/workflows/beam_beam_Playground_CI_Nightly.yml?query=event%3Aschedule) |
| [ Publish Beam SDK Snapshots ](https://github.com/apache/beam/actions/workflows/beam_Publish_Beam_SDK_Snapshots.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_Publish_Beam_SDK_Snapshots.yml?query=event%3Aschedule) |
+| [ Publish BeamMetrics ](https://github.com/apache/beam/actions/workflows/beam_Publish_BeamMetrics.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_Publish_BeamMetrics.yml?query=event%3Aschedule)
| [ Publish Docker Snapshots ](https://github.com/apache/beam/actions/workflows/beam_Publish_Docker_Snapshots.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_Publish_Docker_Snapshots.yml?query=event%3Aschedule) |
+| [ Publish Website ](https://github.com/apache/beam/actions/workflows/beam_Publish_Website.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_Publish_Website.yml?query=event%3Aschedule) |
+| [ Release Nightly Snapshot ](https://github.com/apache/beam/actions/workflows/beam_Release_NightlySnapshot.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_Release_NightlySnapshot.yml?query=event%3Aschedule) |
+| [ Release Nightly Snapshot Python ](https://github.com/apache/beam/actions/workflows/beam_Release_Python_NightlySnapshot.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_Release_Python_NightlySnapshot.yml?query=event%3Aschedule) |
| [ Rotate IO-Datastores Cluster Credentials ](https://github.com/apache/beam/actions/workflows/beam_IODatastoresCredentialsRotation.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_IODatastoresCredentialsRotation.yml?query=event%3Aschedule) |
| [ Rotate Metrics Cluster Credentials ](https://github.com/apache/beam/actions/workflows/beam_MetricsCredentialsRotation.yml) | N/A | [](https://github.com/apache/beam/actions/workflows/beam_MetricsCredentialsRotation.yml?query=event%3Aschedule) |
-| [ Playground CI Nightly ](https://github.com/apache/beam/actions/workflows/beam_Playground_CI_Nightly.yml) | ["java", "go", "python"] | [](https://github.com/apache/beam/actions/workflows/beam_beam_Playground_CI_Nightly.yml?query=event%3Aschedule) |
diff --git a/.github/workflows/beam_PostCommit_Java_Sickbay.yml b/.github/workflows/beam_PostCommit_Java_Sickbay.yml
index 2e96ae63382..5c7318a7bd8 100644
--- a/.github/workflows/beam_PostCommit_Java_Sickbay.yml
+++ b/.github/workflows/beam_PostCommit_Java_Sickbay.yml
@@ -21,7 +21,7 @@ on:
schedule:
- cron: '30 4/6 * * *'
pull_request_target:
- paths: ['release/trigger_all_tests.json', '.github/trigger_files/beam_PostCommit_Java_Sickbay.json']
+ paths: ['.github/trigger_files/beam_PostCommit_Java_Sickbay.json']
workflow_dispatch:
# This allows a subsequently queued workflow run to interrupt previous runs
diff --git a/.github/workflows/beam_PostCommit_Sickbay_Python.yml b/.github/workflows/beam_PostCommit_Sickbay_Python.yml
index 7ebdbcb3841..1a09341adeb 100644
--- a/.github/workflows/beam_PostCommit_Sickbay_Python.yml
+++ b/.github/workflows/beam_PostCommit_Sickbay_Python.yml
@@ -21,7 +21,7 @@ on:
schedule:
- cron: '0 8 * * *'
pull_request_target:
- paths: ['release/trigger_all_tests.json', '.github/trigger_files/beam_PostCommit_Sickbay_Python.json']
+ paths: ['.github/trigger_files/beam_PostCommit_Sickbay_Python.json']
workflow_dispatch:
# This allows a subsequently queued workflow run to interrupt previous runs
diff --git a/.github/workflows/deploy_release_candidate_pypi.yaml b/.github/workflows/deploy_release_candidate_pypi.yaml
index 8dafd520b1b..690442ff7a7 100644
--- a/.github/workflows/deploy_release_candidate_pypi.yaml
+++ b/.github/workflows/deploy_release_candidate_pypi.yaml
@@ -13,11 +13,8 @@ on:
RC:
description: Integer RC version for the release (e.g. 3 for RC3)
required: true
- PYPI_USER:
- description: PyPi username to perform the upload with
- required: true
- PYPI_PASSWORD:
- description: PyPi password to perform the upload with
+ PYPI_API_TOKEN:
+ description: PyPi API token to perform the PyPi upload with
required: true
env:
@@ -32,8 +29,8 @@ jobs:
- name: Mask PyPi id/password
run: |
# Workaround for Actions bug - https://github.com/actions/runner/issues/643
- PYPI_PASSWORD=$(jq -r '.inputs.PYPI_PASSWORD' $GITHUB_EVENT_PATH)
- echo "::add-mask::$PYPI_PASSWORD"
+ PYPI_API_TOKEN=$(jq -r '.inputs.PYPI_API_TOKEN' $GITHUB_EVENT_PATH)
+ echo "::add-mask::$PYPI_API_TOKEN"
- name: Checkout
uses: actions/checkout@v4
- name: Setup environment
@@ -87,4 +84,4 @@ jobs:
echo "Will upload the following files to PyPI:"
ls
- twine upload * -u "${{ github.event.inputs.PYPI_USER }}" -p "${{ github.event.inputs.PYPI_PASSWORD }}"
+ twine upload * -u __token__ -p "${{ github.event.inputs.PYPI_API_TOKEN }}"
diff --git a/contributor-docs/release-guide.md b/contributor-docs/release-guide.md
index e1d0c67e049..7b59351321c 100644
--- a/contributor-docs/release-guide.md
+++ b/contributor-docs/release-guide.md
@@ -37,7 +37,7 @@ complements the foundation-wide guides:
A Beam release consists of the following:
- ASF source zips archived on
- [dist.apache.org](https://dist.apache.org/release/beam) (later archived to
+ [dist.apache.org](https://dist.apache.org/repos/dist/release/beam/) (later archived to
[archive.apache.org](https://archive.apache.org/dist/beam)
- Java jars and poms published to [Maven
Central](https://mvnrepository.com/artifact/org.apache.beam)
diff --git a/sdks/java/harness/build.gradle b/sdks/java/harness/build.gradle
index 70f9c1f298b..2de578cb32c 100644
--- a/sdks/java/harness/build.gradle
+++ b/sdks/java/harness/build.gradle
@@ -49,24 +49,9 @@ applyJavaNature(
automaticModuleName: 'org.apache.beam.fn.harness',
testShadowJar: true,
shadowJarValidationExcludes: [
- "avro/shaded/com/google/**",
- "com/thoughtworks/paranamer/**",
- "junit/**",
- "io/github/classgraph/**",
- "nonapi/io/github/classgraph/**",
- "org/apache/avro/**",
"org/apache/beam/fn/harness/**",
- "org/apache/beam/model/fnexecution/**",
"org/apache/beam/runners/core/**",
- "org/apache/beam/runners/core/construction/**",
- "org/apache/beam/sdk/extensions/avro/**",
"org/apache/beam/sdk/fn/**",
- "org/apache/commons/**",
- "org/checkerframework/**",
- "org/codehaus/jackson/**",
- "org/hamcrest/**",
- "org/junit/**",
- "org/tukaani/xz/**",
],
shadowClosure:
{
diff --git a/sdks/python/apache_beam/examples/cookbook/bigtableio_it_test.py b/sdks/python/apache_beam/examples/cookbook/bigtableio_it_test.py
index 8cc8b3c73a1..98023fbc624 100644
--- a/sdks/python/apache_beam/examples/cookbook/bigtableio_it_test.py
+++ b/sdks/python/apache_beam/examples/cookbook/bigtableio_it_test.py
@@ -174,9 +174,7 @@ class BigtableIOWriteTest(unittest.TestCase):
if self.instance.exists():
self.instance.delete()
- # TODO(https://github.com/apache/beam/issues/29076): Reenable this test
- # once BigTable issues are fixed.
- @pytest.mark.it_postcommit_sickbay
+ @pytest.mark.it_postcommit
def test_bigtable_write(self):
number = self.number
pipeline_args = self.test_pipeline.options_list