You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by yh...@apache.org on 2025/04/16 21:24:16 UTC

(beam) branch useselfhosted updated (e91abebd476 -> ee04c8d1dd6)

This is an automated email from the ASF dual-hosted git repository.

yhu pushed a change to branch useselfhosted
in repository https://gitbox.apache.org/repos/asf/beam.git


 discard e91abebd476 Use self hosted runner to build docker image
     add de2a83f3673 Bump github.com/docker/docker in /sdks (#34483)
     add bebfcbc84bd Add an experiment to make prism server a singleton (#34623)
     add 458d7a72fa5 Fix new flaky postcommit python tests (#34638)
     add 9331662c934 consume flow control settings updates from job settings (#34539)
     add 8de3d3d74f2 [KafkaIO] BUG FIX : fix-ReadFromKafkaViaSDF-bug-shall-set-coder (#34505)
     add 8fb42c7a730 Update CHANGES.md (#34639)
     add 4ead94006b5 Inject SDK-side flattens while handling input/output coder mismatch in flattens. (#34641)
     new ee04c8d1dd6 Use self hosted runner to build docker image

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (e91abebd476)
            \
             N -- N -- N   refs/heads/useselfhosted (ee04c8d1dd6)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../workflows/beam_Publish_Beam_SDK_Snapshots.yml  |   1 -
 CHANGES.md                                         |   4 +-
 .../dataflow/worker/StreamingDataflowWorker.java   |  42 +++---
 .../config/StreamingGlobalConfigHandle.java        |   3 +-
 .../harness/StreamingWorkerStatusPages.java        |  11 +-
 .../worker/windmill/WindmillConnection.java        |  16 ++-
 .../windmill/client/grpc/GrpcDispatcherClient.java |  21 +--
 .../windmill/client/grpc/GrpcWindmillServer.java   |   6 +-
 .../client/grpc/GrpcWindmillStreamFactory.java     |   6 +-
 .../windmill/client/grpc/stubs/ChannelCache.java   |  66 +++++++--
 .../client/grpc/stubs/IsolationChannel.java        |  34 +++--
 .../client/grpc/stubs/WindmillChannelFactory.java  | 119 +---------------
 ...llChannelFactory.java => WindmillChannels.java} | 100 ++++++++++---
 .../grpc/stubs/WindmillStubFactoryFactoryImpl.java |  22 +--
 .../worker/StreamingDataflowWorkerTest.java        |   5 +-
 .../FanOutStreamingEngineWorkerHarnessTest.java    |   6 +-
 .../harness/WindmillStreamSenderTest.java          |   2 +-
 .../client/grpc/GrpcDirectGetWorkStreamTest.java   |   3 +-
 .../client/grpc/GrpcWindmillServerTest.java        |   4 +-
 .../client/grpc/stubs/ChannelCacheTest.java        | 103 +++++++++++++-
 .../windmill/testing/FakeWindmillStubFactory.java  |   4 +-
 runners/prism/java/build.gradle                    |   5 -
 sdks/go.mod                                        |   2 +-
 sdks/go.sum                                        |   4 +-
 .../beam/runners/prism/internal/handlerunner.go    |  67 ++++-----
 .../pkg/beam/runners/prism/internal/preprocess.go  |   3 +
 .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java |  66 +++++----
 .../org/apache/beam/sdk/io/kafka/KafkaIOIT.java    | 157 +++++++++++++++++++++
 .../examples/streaming_wordcount_it_test.py        |   2 +-
 .../apache_beam/io/gcp/bigquery_read_it_test.py    |   3 +
 .../apache_beam/io/gcp/bigquery_write_it_test.py   |   4 +-
 .../apache_beam/io/gcp/gcsio_integration_test.py   |  15 +-
 .../runners/portability/flink_runner_test.py       |   9 +-
 .../portability/fn_api_runner/fn_runner_test.py    |  16 +++
 .../runners/portability/prism_runner.py            |  10 +-
 .../runners/portability/prism_runner_test.py       |  33 +++++
 .../runners/portability/samza_runner_test.py       |   5 +
 .../runners/portability/spark_runner_test.py       |   7 +-
 38 files changed, 674 insertions(+), 312 deletions(-)
 copy runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/{WindmillChannelFactory.java => WindmillChannels.java} (55%)


(beam) 01/01: Use self hosted runner to build docker image

Posted by yh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

yhu pushed a commit to branch useselfhosted
in repository https://gitbox.apache.org/repos/asf/beam.git

commit ee04c8d1dd62ad76e6248e95969fb5a608d74a5a
Author: Yi Hu <ya...@google.com>
AuthorDate: Tue Apr 15 13:52:50 2025 -0400

    Use self hosted runner to build docker image
---
 .../workflows/beam_Publish_Beam_SDK_Snapshots.yml  | 42 +++++++---------------
 1 file changed, 12 insertions(+), 30 deletions(-)

diff --git a/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml b/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml
index 8aead7937cf..5425ac75f23 100644
--- a/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml
+++ b/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml
@@ -52,7 +52,7 @@ jobs:
     if: |
       github.event_name == 'workflow_dispatch' ||
       (github.event_name == 'schedule' && github.repository == 'apache/beam')
-    runs-on: ubuntu-22.04
+    runs-on: [self-hosted, ubuntu-20.04, main]
     timeout-minutes: 160
     name: ${{ matrix.job_name }} (${{ matrix.container_task }})
     strategy:
@@ -61,22 +61,15 @@ jobs:
         job_name: ["beam_Publish_Beam_SDK_Snapshots"]
         job_phrase: ["N/A"]
         container_task:
-          - "go:container"
-          - "java:container:java8"
-          - "java:container:java11"
-          - "java:container:java17"
-          - "java:container:java21"
-          - "java:container:distroless:java17"
-          - "java:container:distroless:java21"
-          - "python:container:py39"
-          - "python:container:py310"
-          - "python:container:py311"
-          - "python:container:py312"
-          - "python:container:distroless:py39"
-          - "python:container:distroless:py310"
-          - "python:container:distroless:py311"
-          - "python:container:distroless:py312"
-          - "java:expansion-service:container"
+          - ":sdks:java:container:java8:docker :sdks:java:container:java11:docker :sdks:java:container:java17:docker :sdks:java:container:java21:docker :sdks:java:container:distroless:java17:docker :sdks:java:container:distroless:java21:docker :sdks:java:expansion-service:container:docker :sdks:go:container:docker"
+          - ":sdks:python:container:py39:docker"
+          - ":sdks:python:container:py310:docker"
+          - ":sdks:python:container:py311:docker"
+          - ":sdks:python:container:py312:docker"
+          - ":sdks:python:container:distroless:py39:docker"
+          - ":sdks:python:container:distroless:py310:docker"
+          - ":sdks:python:container:distroless:py311:docker"
+          - ":sdks:python:container:distroless:py312:docker"
     steps:
       - uses: actions/checkout@v4
       - name: Setup repository
@@ -94,32 +87,21 @@ jobs:
           echo "BEAM_VERSION=${BEAM_VERSION_LINE#*sdk_version=}" >> $GITHUB_ENV
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v1
-      - name: Authenticate on GCP
-        uses: google-github-actions/auth@v2
-        with:
-          service_account: ${{ secrets.GCP_SA_EMAIL }}
-          credentials_json: ${{ secrets.GCP_SA_KEY }}
       - name: Set up Cloud SDK
         uses: google-github-actions/setup-gcloud@v2
       - name: GCloud Docker credential helper
         run: |
           gcloud auth configure-docker ${{ env.docker_registry }}
-      - name: Setup Java environment
-        if: ${{ startsWith(matrix.container_task, 'java') }}
-        uses: ./.github/actions/setup-environment-action
-        with:
-          java-version: 11
       - name: Setup Python environment
-        if: ${{ startsWith(matrix.container_task, 'python') }}
+        if: ${{ startsWith(matrix.container_task, ':sdks:python') }}
         uses: ./.github/actions/setup-environment-action
         with:
           python-version: default
       - name: run Publish Beam SDK Snapshots script
         uses: ./.github/actions/gradle-command-self-hosted-action
         with:
-          gradle-command: :sdks:${{ matrix.container_task }}:docker
+          gradle-command: ${{ matrix.container_task }}
           arguments: |
-            -Pjava11Home=$JAVA_HOME_11_X64 \
             -Pdocker-repository-root=gcr.io/apache-beam-testing/beam-sdk \
             -Pdocker-tag-list=${{ github.sha }},${BEAM_VERSION},latest \
             -Pcontainer-architecture-list=arm64,amd64 \