You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by tv...@apache.org on 2024/05/14 18:58:38 UTC
(beam) branch tvalentyn-gha updated (0a827f31b7a -> 15c077fac15)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git
omit 0a827f31b7a Include ML deps for coverage analysis.
omit 404b57a7fa1 Move ML compatibility tests to postcommits.
omit 13c133176f1 Lint
omit 05451b62a97 Add a subsuite of unit tests for ML package
add fbb3ae1ebef add retries for flaky test (#31247)
add 45c730ad0c7 Update version tag for dataflow dev containers. (#31248)
add 2e5f69c4bb9 Update try-apache-beam-yaml.ipynb (#31257)
add 66da760357c Update yaml to non-beta link (#31210)
add a0c08272d42 Touch trigger files for all relevant ValidatesRunner and PVR suites
add bf347fe898b Sickbay extremely flaky Flink ParDoLifecycleTest
add ef431ec90f2 Reapply "Add Redistribute transform to Java SDK"
add 737be230347 Merge pull request #31156: Reapply "Add Redistribute transform to Java SDK"
add 466516b016c [Python] Use BufferedLogger instead of logger to log in boot.go (#31265)
add 749c128bef1 Disclaim direct runner result can be different with remote runners (#31250)
add c230655a1e5 Implement the hash_words TFT operation (#31249)
add 296c13a291d Avoid side input issue where Latest.globally produces 2 elements in a window (#31263)
add c88a863870c Add a subsuite of unit tests for ML package and remove them from 'coverage' GHA workflow. (#31255)
add 31e81ffe12a add windmill computation state cache, config loaders, and status pages (#31133)
add 459c87af706 Revert "[flink] Add documentation for Flink runner versions updates"
add 557b5bad1b0 Revert "[#30789] Add support for Flink 1.18 (#31062)"
add 0542c83fc1a Un-revert doc fixes
add bcac88b815e Merge pull request #31274 from apache/debugpostrelease
add c7937001981 Beam YAML day zero notebook save file fix (#31266)
add c1d1bd642d1 Fix a typo in skip condition. (#31267)
add 4a15af019cc Touch trigger files for Dataflow ValidatesRunner tests
add 238792d7c65 Reapply "Add direct Dataflow translation of Redistribute transform"
add 99714991b00 Merge pull request #31268: Reapply "Add direct Dataflow translation of Redistribute transform"
add 8869175a887 Fix nullness errors in TestPipeline
add 7f9264ad91d Merge pull request #31230: Fix nullness errors in TestPipeline
add ee170e142b9 Set backlog in gauge metric (#31137)
add 5c5bcccf516 Copy edit the code contribution guide (#31279)
add c061b2776b0 Revert "Don't specify an unnecessary config given that tox task and gradle config already defines the Python version." (#31283)
add 15c077fac15 Don't install TFT on Python 3.11+
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (0a827f31b7a)
\
N -- N -- N refs/heads/tvalentyn-gha (15c077fac15)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../test-properties.json | 2 +-
.../beam_PostCommit_Java_PVR_Flink_Batch.json | 4 +
.../beam_PostCommit_Java_PVR_Flink_Docker.json | 4 +
.../beam_PostCommit_Java_PVR_Flink_Streaming.json | 4 +
.../beam_PostCommit_Java_PVR_Samza.json | 4 +
.../beam_PostCommit_Java_PVR_Spark3_Batch.json | 4 +
.../beam_PostCommit_Java_PVR_Spark3_Streaming.json | 3 +-
...m_PostCommit_Java_ValidatesRunner_Dataflow.json | 4 +-
...Java_ValidatesRunner_Dataflow_JavaVersions.json | 4 +
...it_Java_ValidatesRunner_Dataflow_Streaming.json | 4 +-
...ostCommit_Java_ValidatesRunner_Dataflow_V2.json | 4 +-
...Java_ValidatesRunner_Dataflow_V2_Streaming.json | 4 +-
...eam_PostCommit_Java_ValidatesRunner_Direct.json | 4 +
...t_Java_ValidatesRunner_Direct_JavaVersions.json | 4 +
...beam_PostCommit_Java_ValidatesRunner_Flink.json | 3 +-
...stCommit_Java_ValidatesRunner_Flink_Java11.json | 3 +-
...beam_PostCommit_Java_ValidatesRunner_Samza.json | 3 +-
...beam_PostCommit_Java_ValidatesRunner_Spark.json | 3 +-
...a_ValidatesRunner_SparkStructuredStreaming.json | 3 +-
...stCommit_Java_ValidatesRunner_Spark_Java11.json | 4 +
...m_PostCommit_Java_ValidatesRunner_Twister2.json | 3 +-
.../beam_PostCommit_Java_ValidatesRunner_ULR.json | 3 +-
.github/workflows/README.md | 1 +
.../workflows/beam_PostCommit_Java_Tpcds_Flink.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 2 +-
.github/workflows/beam_PreCommit_Python_ML.yml | 2 +-
CHANGES.md | 1 -
contributor-docs/code-change-guide.md | 439 +++++++-----
.../apache/beam/examples/snippets/Snippets.java | 14 +-
.../get-started/try-apache-beam-yaml.ipynb | 22 +-
gradle.properties | 2 +-
runners/flink/1.18/build.gradle | 25 -
.../flink/1.18/job-server-container/build.gradle | 26 -
runners/flink/1.18/job-server/build.gradle | 31 -
runners/flink/flink_runner.gradle | 5 +-
runners/flink/job-server/flink_job_server.gradle | 2 +
.../dataflow/DataflowPipelineTranslator.java | 37 +
.../beam/runners/dataflow/util/PropertyNames.java | 1 +
.../dataflow/worker/StreamingDataflowWorker.java | 792 +++++++--------------
.../dataflow/worker/status/WorkerStatusPages.java | 4 +
.../worker/streaming/ComputationState.java | 27 +-
.../worker/streaming/ComputationStateCache.java | 218 ++++++
.../worker/streaming/config/ComputationConfig.java | 59 ++
...StreamingApplianceComputationConfigFetcher.java | 136 ++++
.../StreamingEngineComputationConfigFetcher.java | 283 ++++++++
.../config/StreamingEnginePipelineConfig.java | 57 ++
.../streaming/harness/MetricsDataProvider.java | 83 +++
.../harness/StreamingWorkerStatusPages.java | 261 +++++++
.../worker/util/common/worker/WorkExecutor.java | 2 +-
.../worker/windmill/WindmillServerBase.java | 2 +-
.../worker/windmill/WindmillServerStub.java | 3 +-
.../windmill/client/grpc/GrpcDispatcherClient.java | 2 +-
.../windmill/client/grpc/GrpcGetDataStream.java | 2 +-
.../windmill/client/grpc/GrpcWindmillServer.java | 30 +-
.../worker/StreamingDataflowWorkerTest.java | 407 ++++++-----
.../streaming/ComputationStateCacheTest.java | 282 ++++++++
...amingApplianceComputationConfigFetcherTest.java | 142 ++++
...treamingEngineComputationConfigFetcherTest.java | 219 ++++++
runners/portability/java/build.gradle | 2 +
runners/samza/job-server/build.gradle | 2 +
runners/spark/job-server/spark_job_server.gradle | 8 +
runners/spark/spark_runner.gradle | 7 +
sdks/go/examples/stringsplit/stringsplit.go | 2 +-
sdks/go/examples/wasm/README.md | 2 +-
.../org/apache/beam/sdk/testing/TestPipeline.java | 85 ++-
.../apache/beam/sdk/transforms/Redistribute.java | 300 ++++++++
.../util/construction/PTransformTranslation.java | 8 +
.../beam/sdk/transforms/RedistributeTest.java | 410 +++++++++++
.../beam/sdk/io/kafka/KafkaUnboundedReader.java | 18 +-
sdks/python/apache_beam/dataframe/frames_test.py | 2 +-
sdks/python/apache_beam/io/requestresponse_test.py | 58 +-
.../ml/inference/onnx_inference_test.py | 5 +-
sdks/python/apache_beam/ml/transforms/tft.py | 44 ++
sdks/python/apache_beam/ml/transforms/tft_test.py | 77 ++
.../python/apache_beam/options/pipeline_options.py | 2 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
sdks/python/apache_beam/transforms/sql.py | 2 +-
sdks/python/container/boot.go | 4 +-
sdks/python/setup.py | 3 +-
sdks/python/tox.ini | 4 +-
sdks/typescript/src/apache_beam/runners/flink.ts | 2 +-
.../content/en/documentation/runners/direct.md | 5 +
.../site/content/en/documentation/runners/flink.md | 8 +-
.../www/site/content/en/documentation/sdks/yaml.md | 2 +-
84 files changed, 3641 insertions(+), 1124 deletions(-)
create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Flink_Batch.json
create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Flink_Docker.json
create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Flink_Streaming.json
create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Samza.json
create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Spark3_Batch.json
create mode 100644 .github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Dataflow_JavaVersions.json
create mode 100644 .github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Direct.json
create mode 100644 .github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Direct_JavaVersions.json
create mode 100644 .github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Spark_Java11.json
delete mode 100644 runners/flink/1.18/build.gradle
delete mode 100644 runners/flink/1.18/job-server-container/build.gradle
delete mode 100644 runners/flink/1.18/job-server/build.gradle
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/ComputationStateCache.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/ComputationConfig.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingApplianceComputationConfigFetcher.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingEngineComputationConfigFetcher.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingEnginePipelineConfig.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/MetricsDataProvider.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/harness/StreamingWorkerStatusPages.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/ComputationStateCacheTest.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingApplianceComputationConfigFetcherTest.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/streaming/config/StreamingEngineComputationConfigFetcherTest.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Redistribute.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RedistributeTest.java