You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by tv...@apache.org on 2026/05/22 22:19:06 UTC
(beam) branch tvalentyn-gha updated (767c6010ccf -> 8cd9163fc7a)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git
omit 767c6010ccf Loosen GRPC requirements.
add e6fcdd72ed6 [Java][Debezium] Fix NPE in debeziumRecordInstant for DELETE events (#37795)
add 8e0736ab304 Loosen GRPC requirements. (#37817)
add 9829d6d3cf9 Merge pull request #37792: [ErrorProne] Fix AutoValueBoxedValues warnings across the codebase
add edaeae93065 update python container tag (#37812)
add 2ebe33d976a fix non-breaking vulnerabilities (#37826)
add 9f5904b2b59 Support inferring schemas from Python dataclasses (#37728)
add 9d73d769ff0 Only pull license for selected tests and publishing container (#37827)
add cf536ea06d6 Fix 36181 cloudml benchmarks job (#37803)
add a0674a97db6 [ErrorProne] Fix InvalidParam, InvalidInlineTag, InvalidBlockTag, and InvalidLink checks (#37773)
add d6bc507d769 Skip intermediate python wheels on pull request trigger (#37832)
add 630f32ada54 [Drain] Expose drain to dofn processElement and onTimer (#37825)
add 87d7bba644f docs: Clarify JAVA_HOME setup and initial build expectations for all OSes (#37415)
add 892fa0a9357 Fix Run in Colab thumbnails blocked by CSP (#37838)
add e497c1e9b43 Partly Revert #37832 (#37837)
add 0705b4376ec apt-get upgrade in container images to install security fixes. (#37841)
add ab566197fd2 upgrade vllm for vuln (#37843)
add 02d8f56a1e8 [Dataflow Streaming] Remove rolled out Windmill isolated channels flag (#37844)
add 3b41e8b4320 Remove rolled out Windmill separate heartbeat stream flag (#37845)
add b92c436ee64 Use Python 3.11 and cibuildwheel 3.4.0 for wheel builds (#37850)
add d6d2433827d Fix python postcommit 30513 (#37849)
add 5c5c710441c Revert "upgrade vllm for vuln" (#37856)
add 47bab7e7597 [SpannerIO] Add low-latency configuration in Spanner Change Streams (#37718)
add 0f6c88e0d4b Pin zookeeper in expansion service (#37859)
add b9a7585d7f4 [ErrorProne] Enable StaticAssignmentInConstructor check (#37786)
add 3b472aaa9d3 [ErrorProne] Enable UnsafeReflectiveConstructionCast check (#37784)
add 914e425b26d Fix #21317: Replace colons with underscores in window filenames for Windows compatibility (#37821)
add 85c1b88ce5a [Dataflow Streaming] Create a separate option to control commit threads with direct path. (#37848)
add 9292e1d8140 [Dataflow Streaming] Prevent commit threads from sharing commit streams (#37847)
add 487696c9fd1 [Dataflow Logging handler] Change @LazyInit to AtomicBoolean (#37846)
add e1e2c3ba85c [DebeziumIO] Implement startOffset & offset persistence for Kafka (#28248) (#37750)
add 4c95230392c [typescript] - fix rest of vulnerabilities (#37852)
add 39c7b46aecb Regenerate beam container image dependencies to include newer version of grpc. (#37822)
add 4aff88ea91a Revert "[Dataflow Streaming] Prevent commit threads from sharing commit streams" (#37873)
add 65179691f25 Fix maven archetype dependency for io-components
add 41b513e210c Merge pull request #37871 from Abacn/fix-maven-archtype
add 90c2ab28a26 [Gemini] Fix bad-unpacking pyrefly error (#37828)
add 0fb490f927f [ErrorProne] Enable EmptyCatch Error Prone check and document ignored exceptions (#37754)
add babf20c99dc Adds validates runner post commit tests for Java streaming worker on streaming engine (#37872)
add 1007f73c8e4 Update python dev container tag (#37853)
add fa9d19063ba SDF draining in dataflow runner v1 (#37831)
add 8c48120697e Enable error_prone UnnecessaryLongToIntConversion check (#37783)
add fe29172781f Updated URLs in notebooks: Folders changed from `/tour-of-beam/` to (#37876)
add 7b33e1cb50b [Java] Improve GroupIntoBatches display data to show batching parameters (#37865)
add ae6d624c2b6 Add a configuration to wait for session creation (#37625)
add d51177b9cee Add a check to make sure GitHub action workflows are listed in readme (#37883)
add fee107ada1f Fix vLLM Gemma benchmark by updating transformers version
add d48108d6f83 Merge pull request #37874 from aIbrahiim/fix-30644-inference-python-benchmark
add 2c99a01f327 Bump google.golang.org/grpc from 1.78.0 to 1.79.3 in /sdks (#37889)
add 180f3a6bdb5 Attempt to fix flaky GrpcLoggingServiceTest (#37891)
add 153875eaf24 Fix #37833: Use named logger instead of root logger in transforms/util.py (#37857)
add 2209bd60122 Adds bundleFinalizer support to Dataflow non-portable worker. (#37723)
add b3d2b06b2a8 [ErrorProne] Fix UnnecessaryMethodReference Check Findings (#37785)
add c05733a3a4d Handle some cases during infer schema from dataclass (#37855)
add d16f5d18d2a [ErrorProne] Fix EqualsGetClass warnings and enable Error Prone check (#37758)
add 7b065555cd0 Fix flakiness in tests by increasing timeout and changing windowing type. (#37887)
add c10028b2583 Fix UnusedVariable warnings across codebase (#37770)
add 48935a83720 Resolve Error Prone NonCanonicalType warnings across the codebase (#37772)
add 4163dc752ac [Error Prone] Fix Slf4j formatting errors and enable Slf4jFormatShouldBeConst check (#37771)
add 4221266ffab [ErrorProne] enable JavaUtilDate, TimeUnitConversionChecker, InvalidThrows checks (#37791)
add 0afbdf684fb Update Go Version to 1.26.1 (#37897)
add 7620a935280 Fix #37736: Allow composite transforms to use implicit input chaining (#37861)
add 181995db2c6 Upgrade Docker setup actions to version 4 (#37914)
add 5139ca67b64 Sync clickhouse resource manager client version with IO (#37918)
add 0e7dd4f3cf7 Add version to beam-sdks-java-io-components
add 866d6f73ddb Merge pull request #37920 from apache/fix-maven-archetypes
add d9a6159d97f Pin docker actions to sha for now
add 0cd35ec0927 Merge pull request #37922 from apache/users/damccorm/shas
add 982c9bd8044 Pin version for setup-qemu-action (#37924)
add ca3421606ee Add Quick Start section to README (#37909)
add 65a896b9780 Fix #37894: Update yaml doc for composite transform changes (#37901)
add 600b7d9050e Pin gcp actions (#37926)
add 45c2863c0a3 Pin to SHA for GCP GitHub Actions
add 90a7db1c094 Merge pull request #37927 from apache/gcp-actions
add 3316ef93182 [Website] Add missing I/O connectors to built-in connectors table (#37903)
add 7c82e73bb1e [Python][Java] Add support for record headers in WriteToKafka (Fixes #27033) (#37458)
add 79c52affb1d Updates YAML Database transforms to use managed I/O by default (#37923)
add cf3d6ed3e52 [ErrorProne] Enable UnnecessaryParentheses validation and fix all violations (#37913)
add 4b448b069f4 Bump Java dev containers (#37931)
add 7187ff67ee8 Handle Flink 2 job server tag in releases and SDK snapshot (#37939)
add 3aad50b99e8 [Security] Bump ActiveMQ from 5.14.5 to 5.19.2 (#37944)
add 012f825245d [Java] Optimize beam pubsub message to serialized pubsub protobuf message (#37896)
add 8e90c36ff1b [Java] Optimize StreamUtils to avoid copying bytes (#37912)
add b6d8f428e29 Iceberg Add Files (#37701)
add 683ecebe3af Fix typo in arc readme (#37948)
add 7431dea675b [#37930][Runners] Reap child processes in ProcessManager to prevent zombie accumulation (#37932)
add 08685524465 Pin setuptools below 82 for TensorFlow Hub pkg_resources compatibility (#37929)
add 40951024bfe Iceberg AddFiles -- handle FileNotFound exceptions (#37952)
add 97da2d69121 Update changes.md (#37936)
add dcb1b756e7d Add OrderedListState support in Go SDK (#37629)
add 2eb71e989b4 Bump picomatch from 2.2.2 to 2.3.2 in /website/www (#37955)
add f8dbba156db Improve docstring for PGBKCVOperation (#37946)
add 299f71d6957 Address no-access pyrefly error (#37888)
add 0e827993098 Fix flaky SpannerChangeStreamOrderedByTimestampAndTransactionId IT (#37950)
add 661add48ae8 Fix version of docker github action (#37970)
add 4b60dae6c9a [yaml] Iceberg add files yaml transform and test (#37938)
add 512c0faedf8 ClickHouseIO - align the RM with the test containers upgrade (#37971)
add d2aed60042e [Security] Bump PostgreSQL JDBC Driver from 42.2.16 to 42.7.10 (#37941)
add 5a65f6ed00c Exposed source window to AssignContext in window_mapping_fn for side inputs (#36722)
add be7ac47c2d5 Add Public Website documentation for RateLimiter (#37949)
add 030460c7fa1 Filter TestOrderedListState for dataflow runner v2 (#37983)
add 72749a2a2bd Some terraform changes to make arc updates work (#37985)
add 19e7c0f2b17 [yaml] - add bigquery destination test (#37787)
add 97c2faf919d Bump picomatch from 2.3.1 to 2.3.2 in /sdks/typescript (#37978)
add 7bda560ebe8 Update Beam website to release 2.72.0 (#37869)
add dc075f3641b Update managed-io.md for release 2.72.0-RC3.
add 755819493e3 Merge pull request #37915 from apache/updates_managed_io_docs_2.72.0_rc3
add 43f1c49d4ac [yaml] Remove test_ReadFromBigQuery_ExternalJavaProvider_0 deselection (#37986)
add c1755e37a12 Bump handlebars from 4.7.7 to 4.7.9 in /sdks/typescript (#37988)
add c4638a39dc7 Migrate pilot workflows to ubuntu-24.04 self-hosted pools (#37995)
add eaf02846d4a Upgrade Snowflake JDBC client to v4 (#37885)
add 12f00acc067 Revert "Migrate pilot workflows to ubuntu-24.04 self-hosted pools (#37995)" (#37998)
add 7d159c77e6e Implemented ML Pipeline Continuous new table rows RunInference (#37647)
add 01b2f219559 Improve AlloyDB notebook setup instructions by clarifying database creation (#37793)
add c0954b797c4 add batch import yaml IT (#37976)
add 6d3b17db06a Adding release-2.72.0-postrelease to protected branches in .asf.yaml
add 47694885673 Add build-essential to self-hosted ARC runner image (#38000)
add e6c9faaf9d9 Add Iceberg Quickstart page (#37739)
add 71405b81be1 Update default version in republish_released_docker_containers.yml (#38010)
add 6f1f242296f Migrate pilot workflows to ubuntu-24.04 self-hosted runners (#38006)
add 1b523ef1d90 Allow sliding windows + combine with hot key fanout (#37899)
add d9cdd6fdf6d [Python] Ad BigQuery Change History source. (#37724)
add e08e9d56e5e Adjust schedule for long-running Java ValidatesRunner Dataflow Streaming variants (#38001)
add 001fc988a7e use pre-1.7.1 implementation (#38016)
add 4da3e553750 Add Python 3.14 support (#37500)
add 0b6a93f486c [#37994] Fix NullPointerException in Spark Runner with multiple outputs and serialization (#38011)
add f4f422c59df [Integration] Expose length-aware batching in all ModelHandler subclasses (#37945)
add 16aee7998a9 Update Python workflows to Ubuntu-24.04 runners, regenerate requirements files (#38020)
add 1fb308859ef Add GNU time to ARC runner image (#38022)
add f456b70ecfd Add ADK model handler (#37917)
add 24f02bb2841 Remove 3.14 runs from ML Tests (#38023)
add ee02db885b8 [Iceberg AddFiles] Use thread pool to process incoming datafiles (#38018)
add b39a95f5d8d Improve bq cdc scalability (#38015)
add b090e22064e Allow tagged types in with_exception_handling. (#37590)
add d301a673218 Fix lint (#38032)
add 5d97107143b adding new metadata skill
add be364d5f8ed add license.
add 4f473d87f7a Merge pull request #37868 from stankiewicz/branch_skill_metadata
add 1ae923e60d5 Set python 3.14 version where needed (#38030)
add d83275adc61 remove void
add 3249e2e334a outstanding jdbc violation
add fae0219d692 cleanup
add 9ea861238d1 Merge pull request #37902 from stankiewicz/error_prone_voidUsed
add d9fa6082ba8 [BigQueryIO] Fix Storage Read API crash on empty tables in streaming pipelines.
add 3fa8b0ca5f4 Update sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java
add bfdac1aaa90 add member variable to keep track of source split result
add b7c87f23ce8 Merge pull request #38009 from stankiewicz/fix_bq_empty_table
add a05482e92d9 - adds new outputWindowedValue(WV) public interface with implementations - add tests to cover metadata propagation across parDos in single and mulitple stages for direct runner - plumbs metadata propagation, where it was lost previously.
add 36d2b4832f3 fix npe in ReduceRunner..
add 80c7a49e743 find npe
add 7a5c9822318 review feedback - remove duplicated code and unused variables
add 1c53cb26cd1 Merge pull request #37851 from stankiewicz/metadata_propagation
add 24f562d84f6 Downgrade Postgres driver to 42.6.2
add 7fdee986a2a Merge pull request #38024 from Abacn/downgrade-postgres
add 295dc1c57be Move Python PostCommits to Ubuntu-24.04 Runners (#38031)
add ceed48c0781 Fix Portable PreCommit on 3.14 (#38033)
add 090133198e5 Add a FailoverChannel wrapper on top of IsolationChannel to maintain a set of primary and failover channel. (#37840)
add bb8e6bcbab9 added GCP auth to Java Dataflow V2 streaming validates runner (#38027)
add 84a28e7e084 Update Python container workflows for parallelization
add c8e45e79c69 Merge pull request #38034 from jrmccluskey/314container
add f7bdf517ccb Fix vLLM worker (#38008)
add 9baebbbae92 Bump lodash from 4.17.23 to 4.18.1 in /website/www (#38042)
add 303a0768f13 Fix 3.14 distroless build, snapshot publishing (#38048)
add 67259e0f1e4 Add PubSub RateLimiter example (#37992)
add db7c5391fbb Bump requests from 2.32.4 to 2.33.0 in /infra/enforcement (#37975)
add bb438338f45 fix vllm lint (#38051)
add f348cd0c4fd [BigQuery] Handle 409 Conflict in BigqueryClient dataset and table creation
add 47bebc33cad [BigQuery] Handle 409 Conflict in BigqueryClient testing client for dataset and table creation
add a7c65ff32b5 Remove 3.14 ML container reference from snapshot action (#38049)
add f7022b5a25b Update Google BOM to 26.79.0
add 2505e4e7fd9 Merge pull request #38021 from kennknowles/update-google-bom
add 8f9c005a5c0 Moving to 2.74.0-SNAPSHOT on master branch.
add 97359441166 Update CHANGES.md for 2.74.0
add 08857d2210b Use ubuntu 24.04 for PostCommit TransformService Direct
add 5f5b03c0d2a Merge pull request #38054 from apache/fix-transform-service
add 60723cde2c0 Bump Python FnAPI beam-master container (#38041)
add 190898cd743 Avoid cloning live dev venv for yaml expansion in CI (#38063)
add b9b7a6729cd Enable SpannerChangeStreamPlacementTableIT and update IntegrationTestEnv cleanup (#38052)
add d0ec8d1e40d fix MetadataPropagationTest.java
add 5ff3611614b Merge pull request #38065 from apanich/patch-6
add 3cc224fb8f1 increase timeout via posargs (#37989)
add 392b869ef86 Fix py314:mongodbioIT (#38062)
add a03784efc93 Fix BigQueryEnrichmentHandler batch handling for duplicate keys (#38040)
add 5932587d20e Add CHANGES.md note for BigQueryEnrichmentHandler duplicate-key batch fix (#38068)
add 211cd6662d2 Run python_dependency_tests during release validation. (#38070)
add 718ff4ec15d Fix Prism global window side input visibility in GetSideData (#38067)
add c1e44fb0258 portable date java and model changes (#38077)
add 6756b1223d0 Fix type in dask runner tox target
add 3875b217aae Merge pull request #38076 from Abacn/fixdask
add 08caae8e328 Bump node-forge from 1.3.2 to 1.4.0 in /sdks/typescript
add 63823a80d53 Merge pull request #37987: Bump node-forge from 1.3.2 to 1.4.0 in /sdks/typescript
add ee3ea237e89 Use python version argument (#38066)
add 28a0cb14ebd Fix GrpcLoggingServiceTest hang on client error (#38082)
add 5fc49f97e99 Ensure LazyFlinkSourceSplitEnumerator handle splits happens after initialized
add 9fe425ffb56 Merge pull request #38083 from Abacn/fix-flink-race
add f7c280c77d2 add waitForReady to logging stream and control stream to handle sdk starting before the runner. (#37982)
add d4ff738d503 Add explicit warning if cython is not available in setup.py (#37953)
add b05b7f3ccca Moves finalize ids out of WorkItem proto and into AssembledWorkItem class. (#37954)
add d1d546976d3 [Java] Add ByteStringOutputStream.isEmpty() and reduce ByteStringOutputStream.size() calls (#37947)
add 31448481939 Optimize ApproximateQuantiles by removing unused context and adding registrations. (#38085)
add c68f520d273 [Iceberg AddFiles] Allow very large batch import (#38039)
add 37ae1ffc176 [Java Portable SDK] Configure JVM so that it exits upon OutOfMemoryError originating in JVM. (#37764)
add 219cac9fae8 Fix Recommendations AI catalog outputs and sklearn pins for 3.14 (#38069)
add a5f000fca5c Fix concurrency for pull_request_target triggered workflows (#38093)
add 8e40d4450b7 compare argument type in a null-friendly way (#38092)
add 7ae7d248fa8 small dep upgrade to avoid vulns (#38091)
add 532dd85dab8 [ML] Add AnthropicModelHandler for direct Claude inference (#37956)
add 46052504435 Fix NameError when pydot is not installed in Beam Playground (#38074)
add 47031efee65 Make MinRowCountForPageSizeCheck available with ValueProvider (#38114)
add a1f260be744 add AWS to IO expansion service (#38111)
add 923ed03ce5a Fix table-row benchmark argument file paths (#38128)
add 4d4da5723c7 Fix ML flakes (#38088)
add 5ee67c6c0bb Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp (#38112)
add 60c8a23b43d Improve the Javadoc for encode/decode methods in coder methods in coder for better clarity on context behaviour
add 760905140cb Apply spotless formatting
add 18395a2a3cc Fix minimal javadoc wording for coder encode decode
add 88ebf83177f Fix serial warning in starter and example archetype
add 379a7f03acf add PicoWordCount Example (Minimal Beam wordcount pipeline)
add cf2eebe615a Fix formmatting (spotless)
add c632f800abb Remove unintended StarterPipeline changes from PicoWordCount PR
add 78b770b920b Address review: use @Element instead of ProcessContext
add 357b7206f82 Merge pull request #38029 from Subramanya-Veeregowda/pico-wordcount-example
add b296b40e3fc Revert "Fix NameError when pydot is not installed in Beam Playground (#38074)" (#38134)
add bb06d9b7040 Fix flaky ViewTest which would fail encoding null value if there were no elements between early trigger and on-time global window firing.
add 91e1b065749 Merge pull request #38129 from scwhittle/flaky_view
add f938385d471 Skip clean up versioned images (#38019)
add e82fc4713f3 Changes cleanup thread to forceExecute. (#38073)
add 6dd599cf3f6 Pin google-adk to 1.17.0 for YAML/Xlang precommit dependency resolution (#38090)
add b4c45719fa6 Migrate self-hosted workflows from ubuntu-20.04 to ubuntu-24.04 (#38133)
add 09c166b8c5c Improved comments and readability in minimal_wordcount example (#37908)
add 379d2124e12 [ErrorProne] Enable JavaTimeDefaultTimeZone check (#37775)
add dfe6bf351fd fix preCommit Spotless rsync install (#38160)
add e1f02622f36 fixed redis tests with testcontainers (#38157)
add 53f8367adf6 Portable Date Type (Python changes) (#38078)
add d89dcacd7c2 migrate MongoDB IO tests from flapdoodle to Testcontainers (#38161)
add cd384e28db3 fix use configured Python in pullLicenses script (#38162)
add 80c4c7a8dd0 Fix spotless in examples:java:iceberg (#38166)
add 549aa066a90 Add rsync to self-hosted ARC runner Docker image (#38164)
add 4ba599c3ffb [Drain] Expose drain to dofn processElement and onTimer - add missing implementation for SDF and WindowExpiration processContext
add 3f5ce80418b Merge pull request #38119 from stankiewicz/fix_drain
add cd1a0998c52 Support annotationlib in native_type_compatibility (#38188)
add f3bd329bbf0 Update go version to 1.26.2 (#38131)
add 98f5b339bab Add mutable change stream in IntegrateionTestEnv.java for postgresql (#38057)
add 5655186de73 Retry transient NOT_FOUND on Storage API default stream
add a056c5b4991 Merge pull request #38132 from aIbrahiim/fix-gcp-io-default-stream
add 0471bb5e935 Change TriggerState finished bitset coder to a SentinelBitSetCoder (#38139)
add 48a6ceb7a57 [Dataflow Streaming] Fix spammy log during GrpcGetDataStream physical stream half-closing (#38158)
add 77ff67063f5 Install openssh-client
add 0bdc149b9b9 Merge pull request #38195 from apache/fix-flink-container
add e6c0796d2c2 Fix Lint PreCommit
add 56c8de59b13 skip spanner assignment too
add 09bc761f57b remove unused ignore
add 2e1edb08232 Merge pull request #38191 from jrmccluskey/assignmentSkip
add 4ee32cddb4c [Java Harness] Reapply optimization to use Caches.weigh on objects within cached blocks (#37964)
add a60a71e238a [Iceberg AddFiles] Attach name-mapping to table (#38189)
add 9a9d241b341 [ErrorProne] Enable BigDecimalEquals check and fix violations (#38193)
add c7e54f81e57 ErrorProne - enable NarrowCalculation check
add b2e97dee8c2 Merge pull request #38192 from stankiewicz/errorprone_NarrowCalculation
add d6fd0698769 [Iceberg] Support nested column paths in keep/drop configuration (#37516)
add ae536b6f0f8 fixed PythonDocker rerun matrix gating and gh setup (#38199)
add 30690fcf82d ignore test temporarily as enabling metadata propagation may break other tests.
add ff68119b42a Merge pull request #38213 from stankiewicz/ignore_metadata_propagation_test
add 3a19ef136fb Enable error-prone checks: ClassInitializationDeadlock, Finalize, NullableTypeParameter, ThreadPriorityCheck (#37788)
add 1f83bf05321 [yaml]: add kafka parameter explicitly and uncover kafka test (#36603)
add fb48a421441 Increase timeout for BQ early rollout job (#38216)
add 688c692702b Bump pytest from 8.4.2 to 9.0.3 in /sdks/python/container/py310 (#38180)
add c8ba2d875be Fix setup python environment
add 0e8953cb9cb Merge pull request #38217 from apache/fix-flink-cluster
add e71ba541497 Add ExceptionInfo type for with_exception_handling (#38173)
add a7c7786645c Migrate Common Linting Tooling to Rust-Based Packages (#38107)
add 9766c4a126d Bump Pytest Bounds (#38218)
add 9ba9960bc6e Use verbose mode when uploading pypi artifacts (#38223)
add c166bbe066c Fix CVE-2026-34477 (#38210)
add e841064660a Bump protobufjs and google-gax in /sdks/typescript (#38224)
add 621354f992b [ErrorProne] Enable UndefinedEquals check (#38197)
add f300e59fff9 ClickHouseIO - Improve user agent (#37815)
add 4e24a9c69f1 [IcebergIO AddFiles] Apply name mapping at commit time (#38232)
add 86ca83e81dd Exclude 3.14 ML containers from push314 (#38221)
add 1cb2a711aa6 Update Google BOM dependencies. (#38239)
add e5c4c1ed162 [Spark Runner] Prepare Spark 3 structured-streaming to shared base, adopt Flink-style version overrides (#38233)
add 0f48ffcbb27 Bump nltk from 3.9.3 to 3.9.4 in /sdks/python/container/ml/py312 (#38227)
add ac161e8d229 Pin numbus-jose-jwt to 9.37.4 (#38187)
add 00e825e71ea Update hbase version to fix netty CVEs. (#38241)
add ce20f7345b5 Remove widget state data in run_inference_gemma.ipynb (#38248)
add 79fb3600f97 [Python] Expand SDF by default in PortableRunner (#37965)
add 2ee8b6b667d Improve Javadoc for Coder class and clarify Context usage (#38235)
add cacf8f43fc9 Bump lodash from 4.17.23 to 4.18.1 in /sdks/typescript (#38043)
add bc6dffee9d3 Bump flatted from 3.4.1 to 3.4.2 in /sdks/typescript (#37906)
add e696acdfad0 Bump github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream in /sdks (#38101)
add 25370f50b39 Bump github.com/nats-io/nats-server/v2 from 2.12.4 to 2.12.6 in /sdks (#37940)
add b4e1d57901e Install python (#38253)
add 0cdd1b49f3b Use static shared catalog and remove premature FileIO close (#38149)
add a615657d5a4 Allow Python user type pass through Beam SQL (#38206)
add 4151dded54f remove processContext usage across examples (java and kotlin) (#37937)
add 3ea73d2ba60 [ErrorProne] enable AutoValueSubclassLeaked, EscapedEntity, JUnitIncompatibleType, EqualsUnsafeCast (#38256)
add 357862aee37 [runners-flink] Bump FlinkRequiresStableInputTest timeout to 120s (#38273)
add 4cdcf04bd78 Add connectivity type tracking to endpoint consumption logic (#38249)
add 5a5f402d6a2 Implement pluggable Lineage in Java SDK (#36781)
add de1cc7d02bd improve Maven badge readability in README (#37910)
add 2f499a34f2c [runners-flink] Use index-based version comparison in flink_runner.gradle (#38272)
add efd2abfe4e8 [ErrorProne] Update comment in BeamModulePlugin.groovy (#38281)
add a4c7944ea68 Add PreCommit Java Dataflow workflow (#38094) (#38124)
add 6106b308996 [BigQueryIO] Include insert error in log message if TableRow parsing fails (#38231)
add e01f711413f bump beam container versions (#38285)
add d225abf7f30 Bump pytest from 8.4.2 to 9.0.3 in /sdks/python/container/py314 (#38179)
add 7597370249d [Enrichment] Expose max_batch_duration_secs in SQL/BQ handlers (#38243) (#38244)
add a375d6c608d FileBasedSinkTest failure (#38284)
add e126106eaba Fix ML presubmits. (#38289)
add b56d94f4334 Automaticaly update BigQuery table schema when unknown fields are seen. (#38058)
add 448422bde99 Fix data race in JsonPayloadSerializerProvider. (#38265)
add f87ee765642 Go validation runner test script fixes (#38283)
add 11cc79e66dd Allow SpannerIO.readChangeStream() to query & union a list of change stream TVF's (#38167)
add 9caf1a6f5cb Update changes (#38293)
add 106e4493873 Pin transformers/tokenizers for ML tests (#38291)
add 9e1c7a8f210 Fix PreCommit Java Dataflow non-portable runner (#38290)
add 48139dd7333 [IcebergIO] Support sort order on dynamic table creation (#38269)
add 3776a9992dd Increase timeouts (#38301)
add e51fb74dbbf Fix LineagePluginTest on runners other than Direct runner (#38296)
add 29b168035b1 Revert #38256 partially (AutoValueSubclassLeaked)
add 4fc480e2c2b Supress warnings
add b4e2c8ca633 Merge pull request #38292 from apache/fix-hadoop-format
add 2f300cc87b7 update beam python container (#38298)
add 7712cf3a460 feat(ml): add stateless bundle-local size-aware batching and benchmark (#37532)
add 725583ef839 Bump pytest from 6.2.5 to 9.0.3 in /playground/infrastructure
add 43e9ea96713 Merge pull request #38165 from apache/dependabot/pip/playground/infrastructure/pytest-9.0.3
add ac7704f068c Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks
add bfa7fb9406d Merge pull request #38102 from apache/dependabot/go_modules/sdks/github.com/aws/aws-sdk-go-v2/service/s3-1.97.3
add 0d1d2476795 Enhance documentation for stateful conversations (#38247)
add 99a72bce645 Use registered type for Row (#38108)
add 6ca2f4ebfe9 Bump Docker PreCommit timeout, address errant setuptools warning (#38305)
add 61fda334aa8 Update Flink Version Compatibility (#38277)
add 062fab3181c Bump nodemailer from 7.0.11 to 8.0.5 in /scripts/ci/issue-report
add 33d6040b61d Merge pull request #38106 from apache/dependabot/npm_and_yarn/scripts/ci/issue-report/nodemailer-8.0.5
add a00d5ab772b Bump picomatch from 2.3.1 to 2.3.2 in /scripts/ci/pr-bot
add 4c8d8522050 Merge pull request #37979 from apache/dependabot/npm_and_yarn/scripts/ci/pr-bot/picomatch-2.3.2
add 281270cbee2 Refactor StorageAPI BigQuery sink (#38264)
add febef3501f6 add auto gemini review comment workflow (#38309)
add d8d99c4b6ec Set gemini config.yaml to post code review on PR open (#38316)
add 7910b1b3793 Add DiskProvisionedIops/ThroughputMibps options and update Go client libraries (#37377)
add 09febfd5d80 Revert "add auto gemini review comment workflow (#38309)" (#38318)
add 79f03db9e10 Remove mypy configuration (#38266)
add b05fbf4437a Remove Samza runner support (#38263)
add 0739dbbbbd3 Revert "Add DiskProvisionedIops/ThroughputMibps options and update Go client libraries (#37377)" (#38321)
add 329488388fd Bump pytest from 8.4.2 to 9.0.3 in /sdks/python/container/ml/py312 (#38178)
add 70405656d36 Bump requests from 2.32.5 to 2.33.0 in /sdks/python/container/ml/py312 (#37968)
add a61ef0a7372 Bump requests (#37962)
add 28576032d43 Add outstanding parameters to processElement and onTimer signatures (#37933)
add ca5cbda4757 Bump torch from 2.7.1 to 2.8.0 in /sdks/python/container/ml/py312 (#38226)
add 63adaabeb19 Bump keras from 3.12.1 to 3.13.2 in /sdks/python/container/ml/py310 (#38198)
add 541d9bce15f Bump xgrammar from 0.1.21 to 0.1.32 in /sdks/python/container/ml/py312 (#37781)
add 083e57902b1 update gemini review to not run on draft prs (#38333)
add 4678606cc95 [runners-spark] Prep shared base for Spark 4 (#38324)
add e45ab70bf5f fix conversion failures - block was at the wrong nesting level (#38336)
add d956eb1c8c7 Update managed-io.md for release 2.73.0-RC2. (#38276)
add e3249fc5b61 Adding release-2.73.0-postrelease to protected branches in .asf.yaml
add c09cfb652ef Update Beam website to release 2.73.0
add 4217116b09f Update release date
add d492624c59c Update changes
add 64a1510010c Merge pull request #38222 from apache/website-2-73
add 03bd44e1c91 Revert "fix preCommit Spotless rsync install (#38160)" (#38196)
add 2e5a3a0fb5a Bump pytest from 8.4.2 to 9.0.3 in /sdks/python/container/ml/py311 (#38177)
add 0a50f319616 Do not set representsation for pythonsdk_any type when typehint is Any (#38325)
add 50f36b2a963 Bump pytest from 8.4.2 to 9.0.3 in /sdks/python/container/py311 (#38174)
add 4aedd4e72fa Ignore container changes from dependabot (#38344)
add 73985e5b9bd Clean up remaining references to Samza runner (#38326)
add fffa3c04e76 Remove Ubuntu 20.04 runner pools (#38335)
add 62fd95d3ebf Merge pull request #38332: only expand update graph if needed
add 9a5318e9ba1 Cross-build java-extensions-avro with Avro 1.12 (#38109)
add e5308097cc3 Revert "remove processContext usage across examples (java and kotlin)" (#38341)
add dfe97e66820 Update republish_released_docker_containers.yml to 2.73.0 (#38354)
add b0cc4320ddd Mark dynamically generated ml_preprocessing test as no_xdist for precommit (#38353)
add 9fdb366d2ed fix: correct typo "occured" to "occurred" (#38087)
add a4cb67621c6 Fix unhandled exception in KafkaIO SDF (#37449) (#37553)
add 8fc2b6b2071 Move Python PreCommit middle versions to PostCommit (#38347)
add ca70eaa3a36 Add DiskProvisionedIops/ThroughputMibps pipeline options for the Java and GO SDKs and update Go client libraries (#38349)
add 42cfe3497b8 Update go version
add ece9b45f289 Update go version base image
add 2575102c44d Install wget
add bcb0726c07f Install wget
add 792021361bd Merge pull request #38356 from apache/fix-playground-go
add c782e1cf47c Revert "Fix unhandled exception in KafkaIO SDF (#37449) (#37553)" (#38361)
add c5d0ab18133 Bump Java bytecode compatibility version to Java11 (#38267)
add 0884c464058 Optimze away WindmillWatermarkHold::clear when the cached hold is empty (#38297)
add d8d12c73892 [runners-spark] Use robust constructor resolution in EncoderFactory (#38271)
add 49abdcf446a add reshuffle as a first class yaml transform and a test (#38046)
add 5938b312827 Run on ubuntu-24.04
add 802331bd99c Merge pull request #38365 from apache/python-arm
add a14d009f59f Improve logging in boot.go to facilitate future triaging (#38342)
add 4baf3ca7242 Upgrade github action versions (#38202)
add 5c1980fd94b Fix SDF bundle finalization timeout in streaming test (#38287)
add 637231ccc7e Add option to use asyncio for AsyncWrapper (#38262)
add 7d3dbcaacb5 Add TableRowMatchers with strict type-aware equality for BigQuery (#37890)
add a5496c61a0f [ValueKind] Add to model (#38308)
add 010c52f2565 Allow Beam Python GCP extra to resolve with google-cloud-storage 3.x (#38135)
add 268ae1acf5e [IcebergIO] Support hash distribution mode when writing rows (#38061)
add 26ef659b84a bump opentelemetry version.
add df9a0424eac add libraries
add 5223c232606 Model changes to allow OpenTelemetry context propagation spotless. rename fields, cleanup groovy spotless rename method, fix constructor invocation.
add 674873d7a87 Merge pull request #36962 from stankiewicz/model
add dc7b5b0190c [Gemini] Migrate all remaining uses of typing types with built-in equivalents (#38334)
add aa5797f355a Add pipeline hash (#38357)
add 16609ed210b fix error prone. (#38372)
add efe4e941939 extend to yaml (#38371)
add d6b98ba9019 Bump minimatch in /scripts/ci/pr-bot (#37732)
add 21b033c07cf Refactor metadata propagation in ReduceFnRunner to support extensible PipelineMetadata
add 110e75934ba Add missing dependency on model:fn-execution to runners:core-java
add fd571c92f59 document asserts due to new state added
add 79f7db946b1 Fix propagation of metadata
add da391d0e3d7 Merge pull request #38230 from stankiewicz/drain_combiner
add 455075b436f Upgrade to Avro 1.12 (#38373)
add 7235088c490 GCS client library migration in Java SDK - part 3 (#37900)
add 6fc8fdebb23 Fix WindowedValue.of() invocation. (#38380)
add 0bd11fd4631 Go VR Flink test on Flink 2.0 (#37640)
add 3a66beeec82 Bump github.com/apache/thrift from 0.21.0 to 0.23.0 in /sdks (#38383)
add 8e0aa0c76f5 update excluded path to relative path (#38378)
add 1db67c7b550 Bump cloud.google.com/go/bigquery from 1.74.0 to 1.77.0 in /sdks (#38385)
add 400b114a0a2 Add Staged Artifact validations for RunnerV2 (#37974)
add fb272738e57 Bump hashicorp/setup-terraform from 3 to 4 (#38389)
add 259e2556b12 Bump actions/checkout from 4 to 6 (#38393)
add c0414c07bdd Bump crazy-max/ghaction-import-gpg from 6.3.0 to 7.0.0 (#38387)
add b4a1794e055 Bump cloud.google.com/go/bigtable from 1.42.0 to 1.47.0 in /sdks (#38396)
add 712ea7e2fbe Bump docker/login-action from 4.0.0 to 4.1.0 (#38394)
add e533dd1054c Bump docker/build-push-action from 7.0.0 to 7.1.0 (#38391)
add 991a8d5a3af Bump google.golang.org/api from 0.276.0 to 0.278.0 in /sdks (#38395)
add e59af80cbc4 Fix FHIR search method signature mismatch after google.golang.org/api update
add c971ec474a7 Merge pull request #38381 from bambadiouf1/fix-fhir-search-sig
add 034d7bf2190 Bump actions/github-script from 8 to 9 (#38402)
add 032cbe2d436 Bump actions/download-artifact from 7 to 8 (#38401)
add 5bc2d6da706 Bump actions/upload-artifact from 4 to 7 (#38400)
add 0972151ba60 Bump google.golang.org/grpc from 1.80.0 to 1.81.0 in /sdks (#38399)
add 0975d7bc83a Bump github.com/lib/pq from 1.11.1 to 1.12.3 in /sdks (#38398)
add bc900805f84 Bump github.com/nats-io/nats-server/v2 from 2.12.6 to 2.14.0 in /sdks (#38397)
add 2190c9fb001 Bump cloud.google.com/go/datastore from 1.22.0 to 1.23.0 in /sdks (#38392)
add b78e11ee2aa Stabilize StorageApiDataTriggeredSchemaUpdateIT assertion (#38339)
add e9a2b6008aa Consolidate linting and type checking configs into pyproject.toml (#38306)
add b30af9eb79c BatchElements transform for Java SDK (#38369)
add 422f6304af1 [yaml] - Add huggingface model handler (#38110)
add 02ac93ed697 Add DiskProvisionedIops/ThroughputMibps pipeline options for the Python SDK (#38370)
add 22c43bf2bd5 Mitigate test broken after Avro Upgrade due to AVRO-4110 (#38405)
add b7b070790b6 Bump cloud.google.com/go/pubsub from 1.50.1 to 1.50.2 in /sdks (#38415)
add c75b09d73d9 Bump cloud.google.com/go/storage from 1.62.0 to 1.62.1 in /sdks (#38414)
add d273ee8a149 Bump github.com/go-sql-driver/mysql from 1.9.3 to 1.10.0 in /sdks (#38413)
add 878042d19da Bump github.com/aws/aws-sdk-go-v2 from 1.41.5 to 1.41.7 in /sdks (#38416)
add 30e98f7fd52 yaml_transform_test - add some debug logs and increase row count (#38367)
add e1b3769e7fb Fix test (#38418)
add a619a580802 fix rrio teardown executor cleanup path (#38417)
add a9cd0179508 Fix SystemError in _DeferredCall.get() under GC pressure (#38355)
add 955b80aa694 Revert "Revert "Bump com.pswidersk.terraform-plugin from 1.0.0 to 1.1.1 (#325…" (#38420)
add b60082d17d4 Revert "[yaml] - Add huggingface model handler (#38110)" (#38421)
add 81828fd2f0b [Dataflow] Added Portable Runner alias to java runners (#38411)
add a8e7ffab371 Fix race conditions, error recovery, and exit handlers in job servers (#38423)
add 9a6734bbe27 Fix PreCommit Java PVR Prism Loopback workflow (#38431)
add 81830f22515 Bump cloud.google.com/go/profiler from 0.4.3 to 0.6.0 in /sdks (#38436)
add 2e6ff74e11e Bump golang.org/x/sys from 0.43.0 to 0.44.0 in /sdks (#38435)
add 8363afe825f Bump cloud.google.com/go/spanner from 1.88.0 to 1.91.0 in /sdks (#38434)
add 5733cc8115d Fix flaky BigQuery file loads by safely handling concurrent mkdirs (#38426)
add a86f2ec49c8 upgrade test containers and fix issues (#38438)
add 399d9d7ea6b Introduce ValueKind to Java and add to WindowedValue (#38315)
add 13bbd5c4d67 Reduce number of layers for Beam container images (#38440)
add f1bbb637e31 Fix deadlock in AsyncWrapper reset_state() (#38427)
add f01b9ddb05e Make Beartype use the default behavior in is_consistent_with() (#38275)
add 1d008bad91f [runners-spark] Add Spark 4 runner (#38255)
add a72b78125ef Fix thread leak for LOOPBACK workers in external worker pool (#38432)
add fba639a05ff Bump urllib3 from 2.6.3 to 2.7.0 in /sdks/python/container/py312 (#38452)
add 81769cbfd13 Adds a new coder translator for Java SchemaCoder. (#37631)
add 118f2a3c3cf Bump github.com/nats-io/nats.go from 1.51.0 to 1.52.0 in /sdks (#38463)
add fb2b8e8953c Bump actions/upload-artifact from 4 to 7 (#38460)
add abf0c7139d3 Bump github.com/aws/aws-sdk-go-v2/config from 1.32.7 to 1.32.17 in /sdks (#38459)
add 9bee04c92d3 Bump golang.org/x/net from 0.53.0 to 0.54.0 in /sdks (#38462)
add 84a63ed6e85 Bump actions/checkout from 4 to 6 (#38464)
add 83706e1ad38 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#38461)
add 34c6e26112c Make SubprocessServer shared cache purging idempotent (#38455)
add 4d683c0dd13 Exercise Spark 4 tests (#38453)
add fbe61eab131 [Python] Python] Bound the memory used for fnapi outbound data messages and receiving messages. (#38407)
add cf5e517db67 update ruff to pyproject (#38474)
add 1f3d66546e7 Expose StorageWriteApiMaxRequestCallbackWaitTimeSec in BQ storage write. (#38470)
add 89dde7c7a1b Upgrade gcsio to 3.1.16 (#38419)
add fa0eef96567 Add retry in connecting manager in MultiProcessShared (#38456)
add ec65d642ccc [Prism] Fix gRPC deadline exceeded errors during bundle failure by passing errgroup context (#38472)
add e7cb9f77973 [Website] add drain update to docs (#38450)
add 79ed7cc166d Remove legacy processContext usage across and replace it with argument provider (#38366)
add e6fed0a2e08 Bump google.golang.org/api from 0.278.0 to 0.279.0 in /sdks (#38480)
add 7ded0e0ce6f Setup Validates runner tests for Spark4 runner (#38478)
add bd52fce27a3 Moving to 2.75.0-SNAPSHOT on master branch.
add 13d36eb3b19 update port for jdbc test (#38485)
add 632f1e45d16 [runners-spark] Spark 4 follow-up: Announce in CHANGES.md and address remaining review comments from #38255 (#38489)
add 3dbd7c8c35b Sickbay two failed tests due to new schema coder urn. (#38497)
add f74b45af2cd Bump Java dev image for Dataflow (#38496)
add 10304c9cd78 Fix gradle command
add 442d6ff7bf2 Merge pull request #38499 from apache/fix-spark4-batch
add 39632ef2134 Fix iceberg gcs dependency after gcsio 3.0 upgrade (#38488)
add 1d6cdc0b4b6 Isolate tests in multi_process_shared with unique temp path. (#38498)
add 5b5cd657c98 Bump @protobufjs/utf8 from 1.1.0 to 1.1.1 in /sdks/typescript (#38475)
add 9a1a4b170e4 [Iceberg] pin hadoop to 3.3.6 (#38500)
add 7b4ae897cc7 Set torch upper bound
add 45790bbe2a2 Merge pull request #38505 from shunping/set-torch-upper-bound
add 84a21684146 Bump google.golang.org/grpc from 1.81.0 to 1.81.1 in /sdks (#38506)
add 5d2ff20e856 Enable multi-release attribute for Spark job server jar (#38449)
add 6e23bc1ef3a Fix empty stream name encountered in StorageApiFinalizeWrtiesDoFn (#38410)
add 6ae46e3c2d4 Bump jackson_version - Fix GHSA-72hv-8253-57qq (#37969)
add 4ac7598df14 Fix Dataflow archetype missing API client dependency (#38509)
add b2d0ef66aa9 Fix pom involving maven-archetype and opentelemetry (#38518)
add 357fd262211 Revert #37631 and #38497 on HEAD (#38516)
add 0496ea7117b Fix typo in README website description (#38469)
add 4c4a2c11e29 Revert "Fix Dataflow archetype missing API client dependency (#38509)" (#38519)
add 1ca1fafcd62 Fix SubprocessServer cache thread-safety and test isolation (#38501)
add 233ebd8e2bc [Dataflow Streaming] Add a job setting to limit value size in windmill state cache (#38458)
add 2ea46979154 Fix non-unique job names in TextIOReadTest (#38242)
add 2c4d2c6de4d Rename run_pylint.sh (#38307)
add 26c6ec7cb74 add todo (#38517)
add 55eb624e5cd Fix interactive environment clean up failure at atexit. (#38526)
add 79ae776fca7 Fix PipelineOptions deserialization NPE (#38531)
add 0318a2fa701 add yaml agent development skill (#38382)
add 01b09725ec3 Bump protobufjs in /sdks/typescript (#38538)
add c3335c887e6 Make sure session creation happens before starting agent (#38477)
add 91f50ce5108 Remove Java 8 variant of beam_PostCommit_Java_ValidatesRunner_Dataflow_JavaVersions (#38541)
add e83a1d5bad9 Pin tensor_rt digest for PyTorch sentiment Dataflow benchmarks (#38374)
add b1f7115a633 Update Dataflow Dependency Java (#38542)
add 6b55c145853 Update Dataflow Proto (#38540)
add bb0bc772d4e Bump google.golang.org/api from 0.279.0 to 0.280.0 in /sdks (#38563)
add ef703f65aff Bump cloud.google.com/go/datastore from 1.23.0 to 1.24.0 in /sdks (#38564)
add f22400a21a9 [Dataflow Streaming] Add experiment to use trigger state to know if a window is new and optimize reads (#37574)
add 2105eaa4d23 Disable grpc fork support on some test suites. (#38566)
add c33bc972f15 Fix race condition in UserPipelineTracker.clear() and various problems (#38537)
add 582bb452985 [Java SDK] Infer Beam logical types for JSR-310 and UUID fields (#38194)
add ba63482cad5 introduce private method to remove clones (#38245)
add 2375fed9735 Refresh Iceberg partition specs periodically (#38408)
add 290e3721b50 WriteToJson - force num_shards (#38484)
add 4f2411e999e Revert "[Java Portable SDK] Configure JVM so that it exits upon OutOfMemoryEr…" (#38567)
add 32ec9bb0946 Bump com.gradle.common-custom-user-data-gradle-plugin (#38574)
add b9b9e78eeae Bump github.com/nats-io/nats-server/v2 from 2.14.0 to 2.14.1 in /sdks (#38576)
add 930b94cceb6 Fix test hang in subprocess expansion service on port bind failure (#38572)
add ad93dd5ad64 Install wget and use repo token (#38340)
add 65e8b655195 huggingface model handler for yaml - retry (#38451)
add 707b48d4c6d Fix table row inference benchmark using wrong model path (#38569)
add 898c1a8e22a Fix flaky MatrixPowerTest.test_basics by reading all generated shards (#38585)
add 0c9b272c56b Update transform catalogue docs (#38457)
add b6aaf42e92e Initial skeleton for the Delta Lake source (#38571)
add 60620e788f0 Enforce Google Maven Mirror on CI environments (#38586)
add 1c0c02472c7 Adds backlog reporting support for non-fnapi based SDF's. (#38346)
add 2203c8d4469 [Prism] Fix DEADLINE_EXCEEDED errors caused by worker failures (#38523)
add 9d307e559eb Suppress log spams in gcsio 3.0 (#38588)
add 8f6e27117e1 Bump golang.org/x/net from 0.54.0 to 0.55.0 in /sdks (#38595)
add 70d8f7dcfdd Revert "huggingface model handler for yaml - retry (#38451)" (#38601)
add 63501f33968 Fix data_race condition in WindmillStreamSenderTest (#38589)
add ef5f89f0508 [Iceberg] Fix manifest bounds being padded with trailing 0x00 bytes (#38580)
add 489efaa2400 Bump docker/build-push-action from 7.1.0 to 7.2.0 (#38597)
add 889ac54b236 Fix cancelled run handling in flaky test prefetcher (#38579)
add a3e33da209c increase timeout (#38604)
add 978c37542c0 Revert "Bump golangci/golangci-lint-action from 3 to 8 (#36291)" (#38606)
add 590c9c27797 Report source lineage from HadoopFormatIO (#37265)
add 679adb85b14 Pipe Valuekind through DoFn and output builders (#38490)
new 8cd9163fc7a Allow regeneration.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (767c6010ccf)
\
N -- N -- N refs/heads/tvalentyn-gha (8cd9163fc7a)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.agent/skills/adding-new-metadata/SKILL.md | 118 +
.agent/skills/beam-dofn-modernizer/SKILL.md | 207 ++
.agent/skills/python-development/SKILL.md | 8 +-
.agent/skills/runners/SKILL.md | 1 -
.agent/skills/yaml-development/SKILL.md | 107 +
.asf.yaml | 4 +
.gemini/config.yaml | 6 +-
.github/ACTIONS.md | 6 +-
.github/ISSUE_TEMPLATE/bug.yml | 2 +-
.github/ISSUE_TEMPLATE/failing_test.yml | 2 +-
.github/ISSUE_TEMPLATE/feature.yml | 2 +-
.github/ISSUE_TEMPLATE/task.yml | 2 +-
.../gradle-command-self-hosted-action/action.yml | 2 +-
.github/actions/rerun-job-action/action.yml | 6 +-
.../test-properties.json | 14 +-
.../actions/setup-environment-action/action.yml | 4 +-
.github/autolabeler.yml | 1 -
.github/build.gradle | 16 +-
.github/codecov.yml | 1 +
.github/dependabot.yml | 2 +
.../gh-actions-self-hosted-runners/arc/README.md | 6 +-
.../arc/environments/beam.env | 62 +-
.github/gh-actions-self-hosted-runners/arc/gke.tf | 13 +
.github/gh-actions-self-hosted-runners/arc/helm.tf | 10 +-
.../arc/images/Dockerfile | 6 +-
.../arc/variables.tf | 8 +-
.github/issue-rules.yml | 4 +-
.../IO_Iceberg_Integration_Tests.json | 2 +-
.../IO_Iceberg_Integration_Tests_Dataflow.json | 2 +-
...Iceberg_Managed_Integration_Tests_Dataflow.json | 2 +-
.../trigger_files/beam_PostCommit_Go_VR_Flink.json | 2 +-
.../beam_PostCommit_Java_DataflowV1.json | 2 +-
.../beam_PostCommit_Java_Jpms_Flink_Java11.json | 2 +-
...m_PostCommit_Java_ValidatesRunner_Dataflow.json | 4 +-
...ValidatesRunner_Dataflow_Streaming_Engine.json} | 0
...beam_PostCommit_Java_ValidatesRunner_Samza.json | 8 -
...beam_PostCommit_Java_ValidatesRunner_Spark.json | 3 +-
.github/trigger_files/beam_PostCommit_Python.json | 6 +-
...am_PostCommit_Python_ValidatesRunner_Flink.json | 3 +-
...am_PostCommit_Python_ValidatesRunner_Spark.json | 3 +-
...t.json => beam_PostCommit_Python_Versions.json} | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.json | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Direct.json | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.json | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Direct.json | 2 +-
.github/trigger_files/beam_PostCommit_SQL.json | 2 +-
.../trigger_files/beam_PostCommit_XVR_Direct.json | 2 +-
.../trigger_files/beam_PostCommit_XVR_Flink.json | 3 +-
.../trigger_files/beam_PostCommit_XVR_Spark3.json | 4 +-
.../beam_PostCommit_Yaml_Xlang_Direct.json | 2 +-
.../trigger_files/beam_PreCommit_Python_ML.json | 4 +
...t.json => beam_PreCommit_Python_PVR_Flink.json} | 0
.github/workflows/IO_Iceberg_Integration_Tests.yml | 6 +-
.../IO_Iceberg_Integration_Tests_Dataflow.yml | 6 +-
..._Iceberg_Managed_Integration_Tests_Dataflow.yml | 6 +-
.github/workflows/IO_Iceberg_Performance_Tests.yml | 6 +-
.github/workflows/IO_Iceberg_Unit_Tests.yml | 10 +-
.github/workflows/README.md | 50 +-
.github/workflows/assign_milestone.yml | 4 +-
.github/workflows/beam_CancelStaleDataflowJobs.yml | 4 +-
.../workflows/beam_CleanUpDataprocResources.yml | 4 +-
.github/workflows/beam_CleanUpGCPResources.yml | 11 +-
.../workflows/beam_CleanUpPrebuiltSDKImages.yml | 4 +-
.../workflows/beam_CloudML_Benchmarks_Dataflow.yml | 6 +-
.../beam_IODatastoresCredentialsRotation.yml | 4 +-
.../beam_Inference_Python_Benchmarks_Dataflow.yml | 30 +-
.../beam_Infrastructure_PolicyEnforcer.yml | 6 +-
.../beam_Infrastructure_SecurityLogging.yml | 6 +-
.../beam_Infrastructure_ServiceAccountKeys.yml | 6 +-
.../beam_Infrastructure_UsersPermissions.yml | 10 +-
.github/workflows/beam_Java_JMH.yml | 6 +-
.../beam_Java_LoadTests_Combine_Smoke.yml | 4 +-
.../beam_LoadTests_Go_CoGBK_Dataflow_Batch.yml | 4 +-
.../beam_LoadTests_Go_CoGBK_Flink_batch.yml | 8 +-
.../beam_LoadTests_Go_Combine_Dataflow_Batch.yml | 4 +-
.../beam_LoadTests_Go_Combine_Flink_Batch.yml | 10 +-
.../beam_LoadTests_Go_GBK_Dataflow_Batch.yml | 4 +-
.../beam_LoadTests_Go_GBK_Flink_Batch.yml | 8 +-
.../beam_LoadTests_Go_ParDo_Dataflow_Batch.yml | 4 +-
.../beam_LoadTests_Go_ParDo_Flink_Batch.yml | 8 +-
.../beam_LoadTests_Go_SideInput_Dataflow_Batch.yml | 4 +-
.../beam_LoadTests_Go_SideInput_Flink_Batch.yml | 8 +-
.../beam_LoadTests_Java_CoGBK_Dataflow_Batch.yml | 4 +-
...eam_LoadTests_Java_CoGBK_Dataflow_Streaming.yml | 6 +-
...s_Java_CoGBK_Dataflow_V2_Batch_JavaVersions.yml | 4 +-
...va_CoGBK_Dataflow_V2_Streaming_JavaVersions.yml | 4 +-
...s_Java_CoGBK_SparkStructuredStreaming_Batch.yml | 4 +-
.../beam_LoadTests_Java_Combine_Dataflow_Batch.yml | 4 +-
...m_LoadTests_Java_Combine_Dataflow_Streaming.yml | 4 +-
...Java_Combine_SparkStructuredStreaming_Batch.yml | 4 +-
.../beam_LoadTests_Java_GBK_Dataflow_Batch.yml | 4 +-
.../beam_LoadTests_Java_GBK_Dataflow_Streaming.yml | 4 +-
.../beam_LoadTests_Java_GBK_Dataflow_V2_Batch.yml | 4 +-
...LoadTests_Java_GBK_Dataflow_V2_Batch_Java17.yml | 4 +-
...am_LoadTests_Java_GBK_Dataflow_V2_Streaming.yml | 4 +-
...Tests_Java_GBK_Dataflow_V2_Streaming_Java17.yml | 4 +-
.../workflows/beam_LoadTests_Java_GBK_Smoke.yml | 4 +-
...sts_Java_GBK_SparkStructuredStreaming_Batch.yml | 4 +-
.../beam_LoadTests_Java_ParDo_Dataflow_Batch.yml | 4 +-
...eam_LoadTests_Java_ParDo_Dataflow_Streaming.yml | 4 +-
...s_Java_ParDo_Dataflow_V2_Batch_JavaVersions.yml | 4 +-
...va_ParDo_Dataflow_V2_Streaming_JavaVersions.yml | 4 +-
...s_Java_ParDo_SparkStructuredStreaming_Batch.yml | 4 +-
.github/workflows/beam_LoadTests_Java_PubsubIO.yml | 4 +-
.../beam_LoadTests_Python_CoGBK_Dataflow_Batch.yml | 4 +-
...m_LoadTests_Python_CoGBK_Dataflow_Streaming.yml | 4 +-
.../beam_LoadTests_Python_CoGBK_Flink_Batch.yml | 4 +-
...eam_LoadTests_Python_Combine_Dataflow_Batch.yml | 4 +-
...LoadTests_Python_Combine_Dataflow_Streaming.yml | 4 +-
.../beam_LoadTests_Python_Combine_Flink_Batch.yml | 4 +-
...am_LoadTests_Python_Combine_Flink_Streaming.yml | 4 +-
...LoadTests_Python_FnApiRunner_Microbenchmark.yml | 4 +-
.../beam_LoadTests_Python_GBK_Dataflow_Batch.yml | 4 +-
...eam_LoadTests_Python_GBK_Dataflow_Streaming.yml | 4 +-
.../beam_LoadTests_Python_GBK_Flink_Batch.yml | 4 +-
...adTests_Python_GBK_reiterate_Dataflow_Batch.yml | 4 +-
...sts_Python_GBK_reiterate_Dataflow_Streaming.yml | 4 +-
.../beam_LoadTests_Python_ParDo_Dataflow_Batch.yml | 4 +-
...m_LoadTests_Python_ParDo_Dataflow_Streaming.yml | 4 +-
.../beam_LoadTests_Python_ParDo_Flink_Batch.yml | 4 +-
...beam_LoadTests_Python_ParDo_Flink_Streaming.yml | 4 +-
...m_LoadTests_Python_SideInput_Dataflow_Batch.yml | 4 +-
.github/workflows/beam_LoadTests_Python_Smoke.yml | 4 +-
.../workflows/beam_MetricsCredentialsRotation.yml | 4 +-
.github/workflows/beam_Metrics_Report.yml | 6 +-
.../workflows/beam_PerformanceTests_AvroIOIT.yml | 4 +-
.../beam_PerformanceTests_AvroIOIT_HDFS.yml | 4 +-
...PerformanceTests_BigQueryIO_Batch_Java_Avro.yml | 6 +-
...PerformanceTests_BigQueryIO_Batch_Java_Json.yml | 6 +-
..._PerformanceTests_BigQueryIO_Streaming_Java.yml | 6 +-
...eam_PerformanceTests_BiqQueryIO_Read_Python.yml | 4 +-
...formanceTests_BiqQueryIO_Write_Python_Batch.yml | 4 +-
.github/workflows/beam_PerformanceTests_Cdap.yml | 4 +-
.../beam_PerformanceTests_Compressed_TextIOIT.yml | 4 +-
...m_PerformanceTests_Compressed_TextIOIT_HDFS.yml | 4 +-
.../beam_PerformanceTests_HadoopFormat.yml | 4 +-
.github/workflows/beam_PerformanceTests_JDBC.yml | 4 +-
.../workflows/beam_PerformanceTests_Kafka_IO.yml | 4 +-
.../beam_PerformanceTests_ManyFiles_TextIOIT.yml | 4 +-
...am_PerformanceTests_ManyFiles_TextIOIT_HDFS.yml | 4 +-
.../beam_PerformanceTests_MongoDBIO_IT.yml | 4 +-
.../beam_PerformanceTests_ParquetIOIT.yml | 4 +-
.../beam_PerformanceTests_ParquetIOIT_HDFS.yml | 4 +-
...erformanceTests_PubsubIOIT_Python_Streaming.yml | 4 +-
...m_PerformanceTests_SQLBigQueryIO_Batch_Java.yml | 6 +-
.../beam_PerformanceTests_SingleStoreIO.yml | 4 +-
..._PerformanceTests_SpannerIO_Read_2GB_Python.yml | 4 +-
...manceTests_SpannerIO_Write_2GB_Python_Batch.yml | 4 +-
.../beam_PerformanceTests_SparkReceiver_IO.yml | 4 +-
.../beam_PerformanceTests_TFRecordIOIT.yml | 4 +-
.../beam_PerformanceTests_TFRecordIOIT_HDFS.yml | 4 +-
.../workflows/beam_PerformanceTests_TextIOIT.yml | 4 +-
.../beam_PerformanceTests_TextIOIT_HDFS.yml | 4 +-
.../beam_PerformanceTests_TextIOIT_Python.yml | 4 +-
...PerformanceTests_WordCountIT_PythonVersions.yml | 6 +-
.../workflows/beam_PerformanceTests_XmlIOIT.yml | 4 +-
.../beam_PerformanceTests_XmlIOIT_HDFS.yml | 4 +-
.../beam_PerformanceTests_xlang_KafkaIO_Python.yml | 4 +-
.github/workflows/beam_Playground_CI_Nightly.yml | 6 +-
.github/workflows/beam_Playground_Precommit.yml | 11 +-
.github/workflows/beam_PostCommit_Go.yml | 8 +-
.../workflows/beam_PostCommit_Go_Dataflow_ARM.yml | 8 +-
.github/workflows/beam_PostCommit_Go_VR_Flink.yml | 6 +-
.github/workflows/beam_PostCommit_Go_VR_Samza.yml | 85 -
.github/workflows/beam_PostCommit_Go_VR_Spark.yml | 6 +-
.github/workflows/beam_PostCommit_Java.yml | 8 +-
.../beam_PostCommit_Java_Avro_Versions.yml | 8 +-
.../beam_PostCommit_Java_BigQueryEarlyRollout.yml | 10 +-
.../workflows/beam_PostCommit_Java_DataflowV1.yml | 8 +-
.../workflows/beam_PostCommit_Java_DataflowV2.yml | 8 +-
.../beam_PostCommit_Java_Examples_Dataflow.yml | 8 +-
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 10 +-
...beam_PostCommit_Java_Examples_Dataflow_Java.yml | 10 +-
.../beam_PostCommit_Java_Examples_Dataflow_V2.yml | 8 +-
...m_PostCommit_Java_Examples_Dataflow_V2_Java.yml | 10 +-
.../beam_PostCommit_Java_Examples_Direct.yml | 8 +-
.../beam_PostCommit_Java_Examples_Flink.yml | 8 +-
.../beam_PostCommit_Java_Examples_Spark.yml | 8 +-
.../beam_PostCommit_Java_Hadoop_Versions.yml | 20 +-
.../beam_PostCommit_Java_IO_Performance_Tests.yml | 10 +-
.../beam_PostCommit_Java_InfluxDbIO_IT.yml | 6 +-
.../beam_PostCommit_Java_Jpms_Dataflow.yml | 8 +-
...beam_PostCommit_Java_Jpms_Dataflow_Versions.yml | 8 +-
.../workflows/beam_PostCommit_Java_Jpms_Direct.yml | 8 +-
.../beam_PostCommit_Java_Jpms_Direct_Versions.yml | 8 +-
.../beam_PostCommit_Java_Jpms_Flink_Java11.yml | 8 +-
.../beam_PostCommit_Java_Jpms_Spark_Java11.yml | 8 +-
.../beam_PostCommit_Java_Nexmark_Dataflow.yml | 6 +-
.../beam_PostCommit_Java_Nexmark_Dataflow_V2.yml | 6 +-
...am_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml | 6 +-
.../beam_PostCommit_Java_Nexmark_Direct.yml | 6 +-
.../beam_PostCommit_Java_Nexmark_Flink.yml | 6 +-
.../beam_PostCommit_Java_Nexmark_Spark.yml | 6 +-
.../beam_PostCommit_Java_PVR_Flink_Batch.yml | 8 +-
.../beam_PostCommit_Java_PVR_Flink_Streaming.yml | 8 +-
.../beam_PostCommit_Java_PVR_Spark3_Streaming.yml | 8 +-
...l => beam_PostCommit_Java_PVR_Spark4_Batch.yml} | 48 +-
... beam_PostCommit_Java_PVR_Spark4_Streaming.yml} | 28 +-
.../beam_PostCommit_Java_PVR_Spark_Batch.yml | 10 +-
.../beam_PostCommit_Java_SingleStoreIO_IT.yml | 6 +-
.../beam_PostCommit_Java_Tpcds_Dataflow.yml | 6 +-
.../workflows/beam_PostCommit_Java_Tpcds_Flink.yml | 6 +-
.../workflows/beam_PostCommit_Java_Tpcds_Spark.yml | 6 +-
...am_PostCommit_Java_ValidatesRunner_Dataflow.yml | 8 +-
..._Java_ValidatesRunner_Dataflow_JavaVersions.yml | 10 +-
...mit_Java_ValidatesRunner_Dataflow_Streaming.yml | 10 +-
..._ValidatesRunner_Dataflow_Streaming_Engine.yml} | 25 +-
...atesRunner_Dataflow_Streaming_TagEncodingV2.yml | 10 +-
...PostCommit_Java_ValidatesRunner_Dataflow_V2.yml | 8 +-
..._Java_ValidatesRunner_Dataflow_V2_Streaming.yml | 17 +-
...beam_PostCommit_Java_ValidatesRunner_Direct.yml | 8 +-
...it_Java_ValidatesRunner_Direct_JavaVersions.yml | 8 +-
.../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 8 +-
...PostCommit_Java_ValidatesRunner_Flink_Java8.yml | 102 -
.../beam_PostCommit_Java_ValidatesRunner_Samza.yml | 100 -
.../beam_PostCommit_Java_ValidatesRunner_Spark.yml | 8 +-
...eam_PostCommit_Java_ValidatesRunner_Spark4.yml} | 27 +-
...va_ValidatesRunner_SparkStructuredStreaming.yml | 8 +-
...PostCommit_Java_ValidatesRunner_Spark_Java8.yml | 102 -
...am_PostCommit_Java_ValidatesRunner_Twister2.yml | 8 +-
.../beam_PostCommit_Java_ValidatesRunner_ULR.yml | 8 +-
.github/workflows/beam_PostCommit_Javadoc.yml | 8 +-
.../beam_PostCommit_PortableJar_Flink.yml | 8 +-
.../beam_PostCommit_PortableJar_Spark.yml | 8 +-
.github/workflows/beam_PostCommit_Python.yml | 10 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 18 +-
.../beam_PostCommit_Python_Dependency.yml | 10 +-
.../beam_PostCommit_Python_Examples_Dataflow.yml | 12 +-
.../beam_PostCommit_Python_Examples_Direct.yml | 10 +-
.../beam_PostCommit_Python_Examples_Flink.yml | 10 +-
.../beam_PostCommit_Python_Examples_Spark.yml | 10 +-
.../beam_PostCommit_Python_MongoDBIO_IT.yml | 14 +-
.../beam_PostCommit_Python_Nexmark_Direct.yml | 6 +-
.../beam_PostCommit_Python_Portable_Flink.yml | 8 +-
...stCommit_Python_ValidatesContainer_Dataflow.yml | 14 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 14 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 10 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 10 +-
...eam_PostCommit_Python_ValidatesRunner_Samza.yml | 110 -
...eam_PostCommit_Python_ValidatesRunner_Spark.yml | 10 +-
...hon.yml => beam_PostCommit_Python_Versions.yml} | 46 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml | 16 +-
.../beam_PostCommit_Python_Xlang_Gcp_Direct.yml | 10 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 13 +-
.../beam_PostCommit_Python_Xlang_IO_Direct.yml | 10 +-
.github/workflows/beam_PostCommit_SQL.yml | 8 +-
.../beam_PostCommit_TransformService_Direct.yml | 12 +-
.github/workflows/beam_PostCommit_Website_Test.yml | 6 +-
.github/workflows/beam_PostCommit_XVR_Direct.yml | 10 +-
.github/workflows/beam_PostCommit_XVR_Flink.yml | 10 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 10 +-
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 10 +-
..._PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml | 12 +-
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 12 +-
.github/workflows/beam_PostCommit_XVR_Samza.yml | 107 -
.github/workflows/beam_PostCommit_XVR_Spark3.yml | 10 +-
.../beam_PostCommit_Yaml_Xlang_Direct.yml | 10 +-
.../workflows/beam_PostRelease_NightlySnapshot.yml | 4 +-
.../workflows/beam_PreCommit_CommunityMetrics.yml | 8 +-
.../workflows/beam_PreCommit_Flink_Container.yml | 6 +-
.github/workflows/beam_PreCommit_GHA.yml | 6 +-
.github/workflows/beam_PreCommit_Go.yml | 6 +-
.github/workflows/beam_PreCommit_GoPortable.yml | 9 +-
.github/workflows/beam_PreCommit_GoPrism.yml | 6 +-
.github/workflows/beam_PreCommit_ItFramework.yml | 8 +-
.github/workflows/beam_PreCommit_Java.yml | 14 +-
...eCommit_Java_Amazon-Web-Services2_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Amqp_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Azure_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Cassandra_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Cdap_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Clickhouse_IO_Direct.yml | 17 +-
.../beam_PreCommit_Java_Csv_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Datadog_IO_Direct.yml | 10 +-
...it_SQL.yml => beam_PreCommit_Java_Dataflow.yml} | 79 +-
.../beam_PreCommit_Java_Debezium_IO_Direct.yml | 10 +-
...beam_PreCommit_Java_ElasticSearch_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Examples_Dataflow.yml | 9 +-
...eam_PreCommit_Java_Examples_Dataflow_Java21.yml | 11 +-
...Commit_Java_File-schema-transform_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Flink_Versions.yml | 8 +-
.../beam_PreCommit_Java_GCP_IO_Direct.yml | 12 +-
.../beam_PreCommit_Java_Google-ads_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_HBase_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_HCatalog_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Hadoop_IO_Direct.yml | 10 +-
.../workflows/beam_PreCommit_Java_IOs_Direct.yml | 10 +-
.../beam_PreCommit_Java_InfluxDb_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_JDBC_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Jms_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Kafka_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Kudu_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_MongoDb_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Mqtt_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Neo4j_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_PVR_Flink_Batch.yml | 10 +-
.../beam_PreCommit_Java_PVR_Flink_Docker.yml | 8 +-
.../beam_PreCommit_Java_PVR_Prism_Loopback.yml | 10 +-
.../beam_PreCommit_Java_Parquet_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Pulsar_IO_Direct.yml | 12 +-
.../beam_PreCommit_Java_RabbitMq_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Redis_IO_Direct.yml | 10 +-
...am_PreCommit_Java_RequestResponse_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_SingleStore_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Snowflake_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Solace_IO_Direct.yml | 8 +-
.../beam_PreCommit_Java_Solr_IO_Direct.yml | 10 +-
....yml => beam_PreCommit_Java_Spark_Versions.yml} | 29 +-
.../beam_PreCommit_Java_Splunk_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Thrift_IO_Direct.yml | 10 +-
.../beam_PreCommit_Java_Tika_IO_Direct.yml | 10 +-
.../workflows/beam_PreCommit_Kotlin_Examples.yml | 6 +-
.../workflows/beam_PreCommit_Portable_Python.yml | 8 +-
.github/workflows/beam_PreCommit_Prism_Python.yml | 8 +-
.github/workflows/beam_PreCommit_Python.yml | 25 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 20 +-
.github/workflows/beam_PreCommit_PythonDocs.yml | 6 +-
.../workflows/beam_PreCommit_PythonFormatter.yml | 6 +-
.github/workflows/beam_PreCommit_PythonLint.yml | 6 +-
.../workflows/beam_PreCommit_Python_Coverage.yml | 8 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 10 +-
.github/workflows/beam_PreCommit_Python_Dill.yml | 8 +-
.../workflows/beam_PreCommit_Python_Examples.yml | 10 +-
.../beam_PreCommit_Python_Integration.yml | 10 +-
.github/workflows/beam_PreCommit_Python_ML.yml | 8 +-
.../workflows/beam_PreCommit_Python_PVR_Flink.yml | 16 +-
.../workflows/beam_PreCommit_Python_Runners.yml | 10 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 10 +-
.github/workflows/beam_PreCommit_RAT.yml | 6 +-
.github/workflows/beam_PreCommit_SQL.yml | 12 +-
.github/workflows/beam_PreCommit_SQL_Java17.yml | 10 +-
.github/workflows/beam_PreCommit_SQL_Java8.yml | 129 -
.github/workflows/beam_PreCommit_Spotless.yml | 8 +-
.github/workflows/beam_PreCommit_Typescript.yml | 6 +-
.github/workflows/beam_PreCommit_Website.yml | 6 +-
.../workflows/beam_PreCommit_Website_Stage_GCS.yml | 10 +-
.github/workflows/beam_PreCommit_Whitespace.yml | 6 +-
.../beam_PreCommit_Xlang_Generated_Transforms.yml | 8 +-
.../workflows/beam_PreCommit_Yaml_Xlang_Direct.yml | 10 +-
.github/workflows/beam_Prober_CommunityMetrics.yml | 4 +-
.github/workflows/beam_Publish_BeamMetrics.yml | 4 +-
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 13 +-
.../workflows/beam_Publish_Docker_Snapshots.yml | 10 +-
.github/workflows/beam_Publish_Website.yml | 8 +-
.../beam_Python_CostBenchmarks_Dataflow.yml | 4 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 12 +-
.github/workflows/beam_Release_NightlySnapshot.yml | 4 +-
.../beam_Release_Python_NightlySnapshot.yml | 4 +-
.../workflows/beam_StressTests_Java_BigQueryIO.yml | 4 +-
.../workflows/beam_StressTests_Java_BigTableIO.yml | 4 +-
.../workflows/beam_StressTests_Java_KafkaIO.yml | 4 +-
.../workflows/beam_StressTests_Java_PubSubIO.yml | 4 +-
.../workflows/beam_StressTests_Java_SpannerIO.yml | 4 +-
.github/workflows/build_release_candidate.yml | 33 +-
.github/workflows/build_runner_image.yml | 10 +-
.github/workflows/build_wheels.yml | 58 +-
.github/workflows/cancel.yml | 2 +-
.github/workflows/code_completion_plugin_tests.yml | 6 +-
.github/workflows/cut_release_branch.yml | 4 +-
.github/workflows/dask_runner_tests.yml | 8 +-
.../workflows/deploy_release_candidate_pypi.yaml | 6 +-
.github/workflows/finalize_release.yml | 44 +-
.github/workflows/flaky_test_detection.yml | 4 +-
.github/workflows/git_tag_released_version.yml | 2 +-
.github/workflows/go_tests.yml | 4 +-
.github/workflows/issue-tagger.yml | 2 +-
.github/workflows/java_tests.yml | 16 +-
.../beam_CloudML_Benchmarks_Dataflow_arguments.txt | 2 +-
...rch_Sentiment_Batch_DistilBert_Base_Uncased.txt | 3 +
...Sentiment_Streaming_DistilBert_Base_Uncased.txt | 3 +
...chmarks_Dataflow_Table_Row_Inference_Batch.txt} | 36 +-
...hmarks_Dataflow_Table_Row_Inference_Stream.txt} | 38 +-
...Python_Benchmarks_Dataflow_VLLM_Gemma_Batch.txt | 1 +
.github/workflows/local_env_tests.yml | 4 +-
.github/workflows/playground_frontend_test.yml | 2 +-
.github/workflows/pr-bot-new-prs.yml | 2 +-
.github/workflows/pr-bot-pr-updates.yml | 2 +-
.github/workflows/pr-bot-prs-needing-attention.yml | 2 +-
.github/workflows/publish_github_release_notes.yml | 4 +-
.github/workflows/python_dependency_tests.yml | 5 +-
.github/workflows/python_tests.yml | 21 +-
.github/workflows/refresh_looker_metrics.yml | 8 +-
.github/workflows/reportGenerator.yml | 2 +-
.../republish_released_docker_containers.yml | 17 +-
.github/workflows/run_perf_alert_tool.yml | 4 +-
.../workflows/run_rc_validation_go_wordcount.yml | 2 +-
.../run_rc_validation_java_mobile_gaming.yml | 6 +-
.../run_rc_validation_java_quickstart.yml | 4 +-
.../run_rc_validation_python_mobile_gaming.yml | 6 +-
.../workflows/run_rc_validation_python_yaml.yml | 6 +-
.github/workflows/self-assign.yml | 2 +-
.github/workflows/tour_of_beam_backend.yml | 4 +-
.../workflows/tour_of_beam_backend_integration.yml | 2 +-
.github/workflows/tour_of_beam_frontend_test.yml | 2 +-
.github/workflows/typescript_tests.yml | 14 +-
.github/workflows/update_python_dependencies.yml | 15 +-
.pre-commit-config.yaml | 13 +-
.test-infra/BUILD_STATUS.md | 390 ---
.../github/github_runs_prefetcher/code/config.yaml | 7 +-
.../github/github_runs_prefetcher/code/main.py | 2 +-
.test-infra/tools/refresh_looker_metrics.py | 3 +-
.../tools/stale_dataflow_prebuilt_image_cleaner.sh | 4 +-
.../src/main/resources/configuration.yaml | 2 +-
CHANGES.md | 72 +-
CONTRIBUTING.md | 5 +
README.md | 21 +-
build.gradle.kts | 18 +-
buildSrc/build.gradle.kts | 19 +
.../org/apache/beam/gradle/BeamDockerPlugin.groovy | 7 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 241 +-
.../org/apache/beam/gradle/Repositories.groovy | 22 +-
contributor-docs/release-guide.md | 2 +-
dev-support/docker/Dockerfile | 2 +-
examples/java/build.gradle | 26 +
.../examples/iceberg/IcebergBatchWriteExample.java | 15 +-
.../IcebergRestCatalogStreamingWriteExample.java | 2 +-
.../iceberg/snippets/IcebergBeamSchemaAndRow.java | 84 +
.../beam/examples/iceberg/snippets/Quickstart.java | 157 ++
.../beam/examples/SchemaTransformExample.java | 10 +-
.../apache/beam/examples/SqlTransformExample.java | 10 +-
.../beam/examples/ApproximateQuantilesExample.java | 10 +-
.../{TopExample.java => BatchElementsExample.java} | 45 +-
.../beam/examples/CoCombineTransformExample.java | 19 +-
.../apache/beam/examples/CoGroupByKeyExample.java | 10 +-
.../org/apache/beam/examples/CombineExample.java | 10 +-
.../org/apache/beam/examples/CountExample.java | 10 +-
.../apache/beam/examples/CountPerKeyExample.java | 10 +-
.../org/apache/beam/examples/CreateExample.java | 10 +-
.../apache/beam/examples/DebuggingWordCount.java | 15 +-
.../org/apache/beam/examples/DistinctExample.java | 10 +-
.../beam/examples/FlatMapElementsExample.java | 10 +-
.../beam/examples/GroupIntoBatchesExample.java | 10 +-
.../beam/examples/KafkaPassengerCountJson.java | 21 +-
.../org/apache/beam/examples/KafkaStreaming.java | 33 +-
.../apache/beam/examples/KafkaWordCountAvro.java | 11 +-
.../apache/beam/examples/KafkaWordCountJson.java | 11 +-
.../java/org/apache/beam/examples/KeysExample.java | 10 +-
.../org/apache/beam/examples/KvSwapExample.java | 10 +-
.../org/apache/beam/examples/LatestExample.java | 10 +-
.../apache/beam/examples/MapElementsExample.java | 10 +-
.../java/org/apache/beam/examples/MaxExample.java | 10 +-
.../org/apache/beam/examples/MaxPerKeyExample.java | 10 +-
.../java/org/apache/beam/examples/MeanExample.java | 10 +-
.../apache/beam/examples/MeanPerKeyExample.java | 10 +-
.../java/org/apache/beam/examples/MinExample.java | 10 +-
.../org/apache/beam/examples/MinPerKeyExample.java | 10 +-
.../org/apache/beam/examples/PartitionExample.java | 10 +-
.../org/apache/beam/examples/PicoWordCount.java | 56 +
...terSimple.java => RateLimitedPubSubReader.java} | 58 +-
.../apache/beam/examples/RateLimiterSimple.java | 12 +-
.../org/apache/beam/examples/RegexExample.java | 10 +-
.../org/apache/beam/examples/SampleExample.java | 10 +-
.../java/org/apache/beam/examples/SumExample.java | 10 +-
.../org/apache/beam/examples/SumPerKeyExample.java | 10 +-
.../org/apache/beam/examples/ToStringExample.java | 10 +-
.../java/org/apache/beam/examples/TopExample.java | 10 +-
.../org/apache/beam/examples/ValuesExample.java | 10 +-
.../java/org/apache/beam/examples/ViewExample.java | 19 +-
.../org/apache/beam/examples/WindowExample.java | 10 +-
.../beam/examples/complete/AutoComplete.java | 60 +-
.../examples/complete/StreamingWordExtract.java | 16 +-
.../org/apache/beam/examples/complete/TfIdf.java | 77 +-
.../examples/complete/TopWikipediaSessions.java | 32 +-
.../beam/examples/complete/TrafficMaxLaneFlow.java | 32 +-
.../beam/examples/complete/TrafficRoutes.java | 50 +-
.../transforms/DataProtectors.java | 21 +-
.../transforms/io/TokenizationBigQueryIO.java | 7 +-
.../transforms/io/TokenizationBigTableIO.java | 6 +-
.../datatokenization/utils/CsvConverters.java | 115 +-
.../datatokenization/utils/ErrorConverters.java | 30 +-
.../datatokenization/utils/SchemasUtils.java | 5 +-
.../beam/examples/complete/game/GameStats.java | 70 +-
.../examples/complete/game/HourlyTeamScore.java | 6 +-
.../beam/examples/complete/game/LeaderBoard.java | 28 +-
.../examples/complete/game/StatefulTeamScore.java | 26 +-
.../beam/examples/complete/game/UserScore.java | 16 +-
.../complete/game/utils/WriteToBigQuery.java | 22 +-
.../examples/complete/game/utils/WriteToText.java | 21 +-
.../game/utils/WriteWindowedToBigQuery.java | 15 +-
.../complete/kafkatopubsub/KafkaToPubsub.java | 10 +-
.../cookbook/BigQueryStreamingTornadoes.java | 23 +-
.../beam/examples/cookbook/BigQueryTornadoes.java | 18 +-
.../examples/cookbook/CombinePerKeyExamples.java | 14 +-
.../beam/examples/cookbook/FilterExamples.java | 31 +-
.../beam/examples/cookbook/JoinExamples.java | 32 +-
.../beam/examples/cookbook/MaxPerKeyExamples.java | 17 +-
.../cookbook/MinimalBigQueryTornadoes.java | 20 +-
.../beam/examples/cookbook/TriggerExample.java | 47 +-
.../apache/beam/examples/snippets/Snippets.java | 40 +-
.../examples/subprocess/ExampleEchoPipeline.java | 10 +-
.../subprocess/kernel/SubProcessIOFiles.java | 9 +-
.../subprocess/kernel/SubProcessKernel.java | 10 +-
.../subprocess/utils/CallingSubProcessUtils.java | 2 +-
.../beam/examples/subprocess/utils/FileUtils.java | 25 +-
.../apache/beam/examples/WindowedWordCountIT.java | 7 +-
.../java/org/apache/beam/examples/WordCountIT.java | 6 +-
.../org/apache/beam/examples/complete/TfIdfIT.java | 6 +-
.../examples/complete/TopWikipediaSessionsIT.java | 7 +-
.../beam/examples/cookbook/DistinctExampleIT.java | 6 +-
.../beam/examples/cookbook/JoinExamplesIT.java | 6 +-
.../beam/examples/cookbook/TriggerExampleTest.java | 8 +-
.../subprocess/ExampleEchoPipelineTest.java | 10 +-
.../ReadFromTwitterDoFn.java | 5 +-
.../twitterstreamgenerator/TwitterConfig.java | 2 +-
.../twitterstreamgenerator/TwitterConnection.java | 1 +
.../ReadFromTwitterDoFnTest.java | 1 +
.../apache/beam/examples/webapis/ImageRequest.java | 2 +-
.../beam/examples/kotlin/DebuggingWordCount.kt | 12 +-
.../examples/kotlin/cookbook/BigQueryTornadoes.kt | 15 +-
.../kotlin/cookbook/CombinePerKeyExamples.kt | 13 +-
.../examples/kotlin/cookbook/FilterExamples.kt | 30 +-
.../beam/examples/kotlin/cookbook/JoinExamples.kt | 31 +-
.../examples/kotlin/cookbook/MaxPerKeyExamples.kt | 15 +-
.../examples/kotlin/cookbook/TriggerExample.kt | 42 +-
.../beam/examples/kotlin/snippets/Snippets.kt | 13 +-
.../alloydb_product_catalog_embeddings.ipynb | 270 ++-
.../notebooks/beam-ml/run_inference_gemma.ipynb | 698 ------
.../learn_beam_windowing_by_doing.ipynb | 2 +-
.../interactive-overview/getting-started.ipynb | 6 +-
.../reading-and-writing-data.ipynb | 4 +-
gradle.properties | 13 +-
infra/enforcement/requirements.txt | 2 +-
it/clickhouse/build.gradle | 7 +-
.../it/clickhouse/ClickHouseResourceManager.java | 35 +-
.../clickhouse/conditions/ClickHouseRowsCheck.java | 4 +-
.../clickhouse/ClickHouseResourceManagerTest.java | 5 +-
.../beam/it/common/utils/IORedirectUtil.java | 2 +-
.../datadog/conditions/DatadogLogEntriesCheck.java | 4 +-
.../java/org/apache/beam/it/gcp/LoadTestBase.java | 4 +-
.../beam/it/gcp/artifacts/utils/ArtifactUtils.java | 4 +-
.../it/gcp/bigquery/BigQueryResourceManager.java | 42 +-
.../gcp/bigquery/conditions/BigQueryRowsCheck.java | 4 +-
.../it/gcp/dataflow/DefaultPipelineLauncher.java | 2 +-
.../beam/it/gcp/dataflow/DirectRunnerClient.java | 4 +-
.../beam/it/gcp/datastore/DatastoreUtils.java | 4 +-
.../org/apache/beam/it/gcp/pubsub/PubsubUtils.java | 4 +-
.../gcp/pubsub/conditions/PubsubMessagesCheck.java | 4 +-
.../apache/beam/it/gcp/bigquery/BigQueryIOLT.java | 6 +-
.../apache/beam/it/gcp/bigquery/BigQueryIOST.java | 4 +-
.../beam/it/gcp/bigquery/BigQueryStreamingLT.java | 8 +-
.../apache/beam/it/gcp/bigtable/BigTableIOLT.java | 8 +-
.../apache/beam/it/gcp/spanner/SpannerIOLT.java | 4 +-
.../apache/beam/it/gcp/spanner/SpannerIOST.java | 4 +-
.../apache/beam/it/gcp/storage/FileBasedIOLT.java | 6 +-
.../beam/it/jdbc/AbstractJDBCResourceManager.java | 2 +-
.../java/org/apache/beam/it/kafka/KafkaIOLT.java | 12 +-
.../java/org/apache/beam/it/kafka/KafkaIOST.java | 4 +-
.../mongodb/conditions/MongoDBDocumentsCheck.java | 4 +-
.../it/splunk/conditions/SplunkEventsCheck.java | 4 +-
.../runner-concepts/description.md | 97 -
learning/tour-of-beam/terraform/build.gradle.kts | 2 +-
local-env-setup.sh | 4 +-
.../beam/model/fn_execution/v1/beam_fn_api.proto | 26 +
.../org/apache/beam/model/pipeline/v1/schema.proto | 7 +
playground/backend/containers/go/build.gradle | 2 +-
.../cloudbuild/playground_ci_examples.sh | 2 +-
playground/infrastructure/requirements.txt | 2 +-
playground/terraform/build.gradle.kts | 2 +-
.../python-release/python_release_automation.sh | 2 +-
release/src/main/scripts/jenkins_jobs.txt | 115 -
release/src/main/scripts/mass_comment.py | 176 --
runners/core-java/build.gradle | 1 +
.../apache/beam/runners/core/CombinedMetadata.java | 87 +
.../runners/core/CombinedMetadataCombiner.java | 78 +
.../runners/core/LateDataDroppingDoFnRunner.java | 10 +-
.../beam/runners/core/MetadataCombiner.java} | 9 +-
...TimeBoundedSplittableProcessElementInvoker.java | 276 ++-
.../apache/beam/runners/core/ReduceFnRunner.java | 104 +-
.../apache/beam/runners/core/SimpleDoFnRunner.java | 136 +-
.../core/SplittableParDoViaKeyedWorkItems.java | 48 +-
.../core/SplittableProcessElementInvoker.java | 25 +-
.../beam/runners/core/StatefulDoFnRunner.java | 3 +-
.../org/apache/beam/runners/core/StepContext.java | 6 +
.../apache/beam/runners/core/WatermarkHold.java | 17 +
.../construction/SerializablePipelineOptions.java | 2 +-
.../beam/runners/core/metrics/BoundedTrieData.java | 4 +-
.../beam/runners/core/metrics/MetricsLogger.java | 6 +-
.../runners/core/metrics/SimpleExecutionState.java | 2 +-
.../core/serialization/SentinelBitSetCoder.java | 81 +
.../core/triggers/FinishedTriggersBitSet.java | 14 +
.../core/triggers/TriggerStateMachineRunner.java | 47 +-
.../beam/runners/core/CombinedMetadataTest.java | 70 +
.../core/InMemoryMultimapSideInputViewTest.java | 13 +-
.../runners/core/MergingActiveWindowSetTest.java | 6 +-
...BoundedSplittableProcessElementInvokerTest.java | 55 +-
.../beam/runners/core/ReduceFnRunnerTest.java | 58 +
.../apache/beam/runners/core/ReduceFnTester.java | 18 +-
.../SimplePushbackSideInputDoFnRunnerTest.java | 3 +-
.../runners/core/SplittableParDoProcessFnTest.java | 187 +-
.../serialization/SentinelBitSetCoderTest.java | 150 ++
.../triggers/TriggerStateMachineRunnerTest.java | 117 +
.../apache/beam/runners/direct/DirectMetrics.java | 4 +-
.../beam/runners/direct/DirectTimerInternals.java | 4 +-
...LifecycleManagerRemovingTransformEvaluator.java | 2 +-
.../apache/beam/runners/direct/ParDoEvaluator.java | 2 +-
.../beam/runners/direct/WatermarkManager.java | 5 +-
.../direct/BoundedReadEvaluatorFactoryTest.java | 11 +-
.../beam/runners/direct/DirectRegistrarTest.java | 4 +-
...cycleManagerRemovingTransformEvaluatorTest.java | 2 +-
.../runners/direct/DoFnLifecycleManagerTest.java | 2 +-
.../runners/direct/DoFnLifecycleManagersTest.java | 4 +-
.../direct/UnboundedReadEvaluatorFactoryTest.java | 44 +-
.../runners/flink/streaming/StreamSources.java | 2 +-
.../wrappers/streaming/DoFnOperator.java | 4 +-
.../runners/flink/FlinkExecutionEnvironments.java | 14 +-
.../FlinkStreamingPortablePipelineTranslator.java | 5 +-
.../translation/types/CoderTypeInformation.java | 2 +-
.../wrappers/streaming/DoFnOperator.java | 4 +-
.../state/FlinkBroadcastStateInternals.java | 8 +-
.../streaming/state/FlinkStateInternals.java | 16 +-
.../flink/FlinkRequiresStableInputTest.java | 7 +-
.../runners/flink/streaming/StreamSources.java | 2 +-
runners/flink/flink_runner.gradle | 22 +-
.../flink_job_server_container.gradle | 3 +
.../runners/flink/FlinkExecutionEnvironments.java | 10 +-
.../flink/FlinkPortableClientEntryPoint.java | 3 +-
.../flink/FlinkStreamingPipelineTranslator.java | 2 +-
.../FlinkStreamingPortablePipelineTranslator.java | 5 +-
.../runners/flink/metrics/MetricsAccumulator.java | 1 +
.../translation/types/CoderTypeInformation.java | 2 +-
.../translation/types/CoderTypeSerializer.java | 2 +-
.../wrappers/streaming/DoFnOperator.java | 4 +-
.../streaming/ExecutableStageDoFnOperator.java | 4 +-
.../io/source/LazyFlinkSourceSplitEnumerator.java | 17 +-
.../unbounded/FlinkUnboundedSourceReader.java | 2 +-
.../state/FlinkBroadcastStateInternals.java | 8 +-
.../streaming/state/FlinkStateInternals.java | 16 +-
.../flink/FlinkRequiresStableInputTest.java | 15 +-
.../beam/runners/flink/ReadSourcePortableTest.java | 2 +
.../flink/metrics/FlinkMetricContainerTest.java | 8 +-
.../flink/streaming/GroupByWithNullValuesTest.java | 1 +
.../wrappers/streaming/FlinkKeyUtilsTest.java | 3 +-
.../wrappers/streaming/WindowDoFnOperatorTest.java | 10 +
runners/google-cloud-dataflow-java/build.gradle | 21 +-
.../beam/runners/dataflow/BatchViewOverrides.java | 12 +
.../beam/runners/dataflow/DataflowPipelineJob.java | 1 +
.../dataflow/DataflowPipelineTranslator.java | 7 +
.../beam/runners/dataflow/DataflowRunner.java | 27 +-
.../runners/dataflow/GroupIntoBatchesOverride.java | 12 +
.../dataflow/RedistributeByKeyOverrideFactory.java | 1 +
.../runners/dataflow/internal/CustomSources.java | 2 +-
.../options/DataflowPipelineWorkerPoolOptions.java | 14 +
.../options/DataflowStreamingPipelineOptions.java | 26 +-
.../beam/runners/dataflow/util/MonitoringUtil.java | 1 +
.../beam/runners/dataflow/util/PackageUtil.java | 1 +
.../dataflow/DataflowPipelineRegistrarTest.java | 4 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 39 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 167 +-
.../dataflow/PrimitiveParDoSingleFactoryTest.java | 2 +-
.../options/DataflowPipelineOptionsTest.java | 9 +
.../runners/dataflow/util/CloudObjectsTest.java | 4 +-
.../google-cloud-dataflow-java/worker/build.gradle | 1 +
.../dataflow/worker/ApplianceShuffleReader.java | 1 +
.../dataflow/worker/ApplianceShuffleWriter.java | 1 +
.../dataflow/worker/BatchDataflowWorker.java | 1 +
.../beam/runners/dataflow/worker/ConcatReader.java | 2 +-
.../worker/DataflowElementExecutionTracker.java | 8 +-
.../dataflow/worker/DataflowExecutionContext.java | 6 +-
.../dataflow/worker/DoFnInstanceManagers.java | 10 +-
.../MetricsToPerStepNamespaceMetricsConverter.java | 4 +-
.../beam/runners/dataflow/worker/PubsubSink.java | 9 +-
.../worker/SplittableProcessFnFactory.java | 6 +-
.../dataflow/worker/StreamingDataflowWorker.java | 113 +-
.../worker/StreamingModeExecutionContext.java | 90 +-
.../dataflow/worker/UngroupedWindmillReader.java | 19 +-
.../dataflow/worker/WindmillKeyedWorkItem.java | 13 +-
.../beam/runners/dataflow/worker/WindmillSink.java | 29 +-
.../dataflow/worker/WindmillValueKindHelper.java | 60 +
.../runners/dataflow/worker/WorkUnitClient.java | 2 +-
.../dataflow/worker/WorkerCustomSources.java | 2 +-
.../worker/WorkerPipelineOptionsFactory.java | 4 +-
.../logging/DataflowWorkerLoggingHandler.java | 10 +-
.../dataflow/worker/status/DebugCapture.java | 2 +
.../StreamingEngineComputationConfigFetcher.java | 4 +-
.../FanOutStreamingEngineWorkerHarness.java | 42 +-
.../harness/SingleSourceWorkerHarness.java | 10 +
.../dataflow/worker/util/SimpleByteHistogram.java} | 30 +-
.../dataflow/worker/util/ValueInEmptyWindows.java | 12 +
.../worker/util/common/worker/ShuffleEntry.java | 8 +-
.../worker/windmill/WindmillEndpoints.java | 23 +
.../worker/windmill/WindmillServerBase.java | 1 +
.../windmill/client/AbstractWindmillStream.java | 8 +-
.../client/ResettableThrowingStreamObserver.java | 17 +-
.../client/getdata/ApplianceGetDataClient.java | 4 +-
.../client/grpc/GetWorkResponseChunkAssembler.java | 14 +-
.../client/grpc/GrpcDirectGetWorkStream.java | 1 +
.../windmill/client/grpc/GrpcDispatcherClient.java | 50 +-
.../windmill/client/grpc/GrpcGetDataStream.java | 5 +-
.../windmill/client/grpc/GrpcGetWorkStream.java | 1 +
.../client/grpc/GrpcGetWorkerMetadataStream.java | 18 +-
.../windmill/client/grpc/GrpcWindmillServer.java | 3 +-
.../client/grpc/GrpcWindmillStreamFactory.java | 6 +-
.../grpc/observers/DirectStreamObserver.java | 2 +-
.../client/grpc/stubs/FailoverChannel.java | 443 ++++
.../grpc/stubs/WindmillStubFactoryFactory.java | 2 +-
.../grpc/stubs/WindmillStubFactoryFactoryImpl.java | 19 +-
.../worker/windmill/state/WindmillMultimap.java | 4 +-
.../worker/windmill/state/WindmillStateCache.java | 122 +-
.../windmill/state/WindmillWatermarkHold.java | 100 +-
.../worker/windmill/work/WorkItemReceiver.java | 1 +
.../worker/windmill/work/WorkItemScheduler.java | 2 +
.../work/processing/StreamingCommitFinalizer.java | 156 +-
.../work/processing/StreamingWorkScheduler.java | 13 +-
.../work/refresh/StreamPoolHeartbeatSender.java | 27 -
.../worker/CombineValuesFnFactoryTest.java | 2 +-
.../dataflow/worker/DefaultParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/DoFnInstanceManagersTest.java | 3 +-
.../dataflow/worker/FakeWindmillServer.java | 1 +
.../runners/dataflow/worker/ShuffleSinkTest.java | 3 +-
.../runners/dataflow/worker/SimpleParDoFnTest.java | 111 +-
.../worker/StreamingDataflowWorkerTest.java | 174 +-
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 6 +
.../worker/StreamingModeExecutionContextTest.java | 25 +
.../dataflow/worker/TestShuffleReaderTest.java | 4 +-
.../worker/UngroupedShuffleReaderTest.java | 2 +-
.../dataflow/worker/WindmillKeyedWorkItemTest.java | 43 +
.../dataflow/worker/WorkerCustomSourcesTest.java | 13 +-
.../logging/DataflowWorkerLoggingHandlerTest.java | 1 +
.../config/FixedGlobalConfigHandleTest.java | 5 +-
.../StreamingGlobalConfigHandleImplTest.java | 17 +-
.../FanOutStreamingEngineWorkerHarnessTest.java | 162 +-
.../harness/WindmillStreamSenderTest.java | 1 +
.../worker/util/SimpleByteHistogramTest.java | 34 +-
.../worker/CachingShuffleBatchReaderTest.java | 2 +-
.../client/grpc/GrpcDirectGetWorkStreamTest.java | 5 +
.../client/grpc/GrpcDispatcherClientTest.java | 141 --
.../grpc/GrpcGetWorkerMetadataStreamTest.java | 40 -
.../client/grpc/GrpcWindmillServerTest.java | 4 +-
.../client/grpc/stubs/FailoverChannelTest.java | 575 +++++
.../client/grpc/stubs/IsolationChannelTest.java | 64 +-
.../windmill/client/grpc/stubs/NoopClientCall.java | 72 +
.../windmill/state/WindmillStateCacheTest.java | 62 +
.../windmill/state/WindmillStateInternalsTest.java | 206 +-
.../testing/FakeWindmillStubFactoryFactory.java | 2 +-
.../processing/StreamingCommitFinalizerTest.java | 188 ++
.../refresh/StreamPoolHeartbeatSenderTest.java | 84 -
.../worker/windmill/src/main/proto/windmill.proto | 29 +-
.../artifact/ArtifactStagingService.java | 2 -
.../control/DefaultJobBundleFactory.java | 4 +
.../control/FnApiControlClientPoolService.java | 2 +-
...renceCountingExecutableStageContextFactory.java | 2 +-
.../fnexecution/environment/ProcessManager.java | 20 +-
.../fnexecution/logging/Slf4jLogWriter.java | 1 +
.../fnexecution/status/WorkerStatusClient.java | 6 +-
.../artifact/ArtifactStagingServiceTest.java | 3 +-
.../control/DefaultJobBundleFactoryTest.java | 2 +-
.../fnexecution/control/RemoteExecutionTest.java | 4 +-
.../logging/GrpcLoggingServiceTest.java | 117 +-
.../runners/fnexecution/wire/CommonCoderTest.java | 2 +-
.../runners/jobsubmission/InMemoryJobService.java | 26 +-
.../beam/runners/jobsubmission/JobInvocation.java | 3 +-
.../PortablePipelineJarCreatorTest.java | 4 +-
.../apache/beam/runners/jet/JetPipelineResult.java | 2 +-
.../org/apache/beam/runners/jet/JetRunner.java | 2 +-
.../java/org/apache/beam/runners/jet/Utils.java | 2 +-
.../apache/beam/runners/prism/PrismExecutor.java | 6 +-
.../beam/runners/prism/PrismPipelineResult.java | 2 +-
.../beam/runners/prism/PrismRegistrarTest.java | 4 +-
runners/samza/build.gradle | 194 --
runners/samza/job-server/build.gradle | 253 --
.../beam/runners/samza/SamzaExecutionContext.java | 72 -
.../runners/samza/SamzaExecutionEnvironment.java | 44 -
.../apache/beam/runners/samza/SamzaJobInvoker.java | 82 -
.../beam/runners/samza/SamzaJobServerDriver.java | 101 -
.../samza/SamzaPipelineLifeCycleListener.java | 44 -
.../beam/runners/samza/SamzaPipelineOptions.java | 182 --
.../samza/SamzaPipelineOptionsValidator.java | 54 -
.../beam/runners/samza/SamzaPipelineResult.java | 161 --
.../beam/runners/samza/SamzaPipelineRunner.java | 89 -
.../samza/SamzaPortablePipelineOptions.java | 32 -
.../runners/samza/SamzaPortablePipelineResult.java | 48 -
.../org/apache/beam/runners/samza/SamzaRunner.java | 213 --
.../runners/samza/SamzaRunnerOverrideConfigs.java | 85 -
.../beam/runners/samza/SamzaRunnerRegistrar.java | 53 -
.../apache/beam/runners/samza/TestSamzaRunner.java | 109 -
.../runners/samza/adapter/BoundedSourceSystem.java | 449 ----
.../samza/adapter/UnboundedSourceSystem.java | 533 -----
.../samza/container/BeamContainerRunner.java | 89 -
.../samza/container/BeamJobCoordinatorRunner.java | 78 -
.../samza/container/ContainerCfgLoader.java | 62 -
.../samza/container/ContainerCfgLoaderFactory.java | 30 -
.../samza/metrics/DoFnRunnerWithMetrics.java | 108 -
.../samza/metrics/FnWithMetricsWrapper.java | 50 -
.../runners/samza/metrics/SamzaGBKMetricOp.java | 194 --
.../beam/runners/samza/metrics/SamzaMetricOp.java | 173 --
.../samza/metrics/SamzaMetricOpFactory.java | 73 -
.../samza/metrics/SamzaMetricsContainer.java | 120 -
.../metrics/SamzaTransformMetricRegistry.java | 208 --
.../samza/metrics/SamzaTransformMetrics.java | 142 --
.../runners/samza/runtime/AsyncDoFnRunner.java | 188 --
.../beam/runners/samza/runtime/BundleManager.java | 90 -
.../samza/runtime/ClassicBundleManager.java | 329 ---
.../apache/beam/runners/samza/runtime/DoFnOp.java | 579 -----
.../runtime/DoFnRunnerWithKeyedInternals.java | 126 -
.../runners/samza/runtime/FutureCollector.java | 67 -
.../runners/samza/runtime/FutureCollectorImpl.java | 99 -
.../beam/runners/samza/runtime/GroupByKeyOp.java | 243 --
.../beam/runners/samza/runtime/KeyedInternals.java | 180 --
.../beam/runners/samza/runtime/KeyedTimerData.java | 216 --
.../runners/samza/runtime/KvToKeyedWorkItemOp.java | 37 -
.../org/apache/beam/runners/samza/runtime/Op.java | 64 -
.../beam/runners/samza/runtime/OpAdapter.java | 237 --
.../beam/runners/samza/runtime/OpMessage.java | 161 --
.../samza/runtime/OutputManagerFactory.java | 31 -
.../samza/runtime/PortableBundleManager.java | 218 --
.../beam/runners/samza/runtime/PortableDoFnOp.java | 467 ----
.../runners/samza/runtime/SamzaAssignContext.java | 56 -
.../runners/samza/runtime/SamzaDoFnRunners.java | 506 ----
.../SamzaExecutableStageContextFactory.java | 61 -
.../runtime/SamzaMetricsBundleProgressHandler.java | 158 --
.../samza/runtime/SamzaStateRequestHandlers.java | 179 --
.../samza/runtime/SamzaStoreStateInternals.java | 1131 ---------
.../samza/runtime/SamzaTimerInternalsFactory.java | 733 ------
.../samza/runtime/SingletonKeyedWorkItem.java | 49 -
.../SplittableParDoProcessKeyedElementsOp.java | 248 --
.../beam/runners/samza/runtime/WindowAssignOp.java | 53 -
.../beam/runners/samza/runtime/package-info.java | 20 -
.../beam/runners/samza/state/SamzaMapState.java | 40 -
.../beam/runners/samza/state/SamzaSetState.java | 39 -
.../beam/runners/samza/state/package-info.java | 20 -
.../samza/transforms/GroupWithoutRepartition.java | 60 -
.../runners/samza/transforms/package-info.java | 20 -
.../runners/samza/translation/ConfigBuilder.java | 360 ---
.../runners/samza/translation/ConfigContext.java | 79 -
.../translation/FlattenPCollectionsTranslator.java | 111 -
.../samza/translation/GroupByKeyTranslator.java | 272 ---
.../samza/translation/ImpulseTranslator.java | 80 -
.../runners/samza/translation/PViewToIdMapper.java | 83 -
.../translation/ParDoBoundMultiTranslator.java | 563 -----
.../translation/PortableTranslationContext.java | 117 -
.../runners/samza/translation/ReadTranslator.java | 91 -
.../translation/RedistributeByKeyTranslator.java | 64 -
.../samza/translation/ReshuffleTranslator.java | 134 --
.../translation/SamzaImpulseSystemFactory.java | 146 --
.../samza/translation/SamzaPipelineTranslator.java | 211 --
.../SamzaPortablePipelineTranslator.java | 112 -
.../SamzaPortableTranslatorRegistrar.java | 25 -
.../samza/translation/SamzaPublishView.java | 66 -
.../SamzaPublishViewTransformOverride.java | 66 -
.../translation/SamzaPublishViewTranslator.java | 76 -
.../translation/SamzaTestStreamSystemFactory.java | 179 --
.../translation/SamzaTestStreamTranslator.java | 148 --
.../samza/translation/SamzaTransformOverrides.java | 62 -
.../translation/SplittableParDoTranslators.java | 157 --
.../runners/samza/translation/StateIdParser.java | 69 -
.../translation/TransformConfigGenerator.java | 40 -
.../samza/translation/TransformTranslator.java | 45 -
.../samza/translation/TranslationContext.java | 379 ---
.../samza/translation/WindowAssignTranslator.java | 80 -
.../runners/samza/translation/package-info.java | 20 -
.../beam/runners/samza/util/ConfigUtils.java | 43 -
.../apache/beam/runners/samza/util/DoFnUtils.java | 75 -
.../beam/runners/samza/util/FutureUtils.java | 67 -
.../beam/runners/samza/util/HashIdGenerator.java | 66 -
.../runners/samza/util/PipelineJsonRenderer.java | 335 ---
.../runners/samza/util/PortableConfigUtils.java | 43 -
.../beam/runners/samza/util/SamzaCoders.java | 77 -
.../samza/util/SamzaPipelineExceptionListener.java | 34 -
.../samza/util/SamzaPipelineTranslatorUtils.java | 61 -
.../apache/beam/runners/samza/util/StateUtils.java | 39 -
.../beam/runners/samza/util/StoreIdGenerator.java | 48 -
.../beam/runners/samza/util/WindowUtils.java | 74 -
.../beam/runners/samza/util/package-info.java | 20 -
runners/samza/src/main/resources/log4j.properties | 23 -
.../samza/src/main/resources/samza-conf.properties | 37 -
.../samza/SamzaPipelineOptionsValidatorTest.java | 60 -
.../samza/adapter/BoundedSourceSystemTest.java | 309 ---
.../runners/samza/adapter/TestBoundedSource.java | 188 --
.../runners/samza/adapter/TestCheckpointMark.java | 40 -
.../runners/samza/adapter/TestSourceHelpers.java | 162 --
.../runners/samza/adapter/TestUnboundedSource.java | 208 --
.../samza/adapter/UnboundedSourceSystemTest.java | 405 ----
.../TestSamzaRunnerWithTransformMetrics.java | 323 ---
.../metrics/TestSamzaTransformMetricsRegistry.java | 191 --
.../runners/samza/runtime/AsyncDoFnRunnerTest.java | 240 --
.../samza/runtime/ClassicBundleManagerTest.java | 457 ----
.../samza/runtime/FutureCollectorImplTest.java | 92 -
.../runners/samza/runtime/GroupByKeyOpTest.java | 133 --
.../runners/samza/runtime/KeyedTimerDataTest.java | 65 -
.../samza/runtime/PortableBundleManagerTest.java | 178 --
.../SamzaMetricsBundleProgressHandlerTest.java | 187 --
.../runtime/SamzaStoreStateInternalsTest.java | 432 ----
.../runtime/SamzaTimerInternalsFactoryTest.java | 751 ------
.../samza/runtime/SdkHarnessDoFnRunnerTest.java | 48 -
.../samza/translation/ConfigGeneratorTest.java | 461 ----
.../samza/translation/SamzaImpulseSystemTest.java | 65 -
.../samza/translation/TranslationContextTest.java | 98 -
.../beam/runners/samza/util/DoFnUtilsTest.java | 84 -
.../beam/runners/samza/util/FutureUtilsTest.java | 107 -
.../samza/util/InMemoryMetricsReporter.java | 49 -
.../samza/util/PipelineJsonRendererTest.java | 146 --
.../samza/util/PortableConfigUtilsTest.java | 58 -
.../runners/samza/util/TestHashIdGenerator.java | 85 -
.../beam/runners/samza/util/WindowUtilsTest.java | 87 -
runners/samza/src/test/resources/ExpectedDag.json | 373 ---
.../samza/src/test/resources/log4j-test.properties | 26 -
runners/spark/3/build.gradle | 2 +-
runners/spark/4/README.md | 73 +
runners/spark/4/build.gradle | 63 +
.../4/job-server/build.gradle} | 16 +-
.../io/BoundedDatasetFactory.java | 13 +-
.../batch/CombinePerKeyTranslatorBatch.java | 6 +-
.../translation/batch/GroupByKeyHelpers.java | 8 +-
.../batch/GroupByKeyTranslatorBatch.java | 78 +-
.../translation/helpers/EncoderFactory.java | 333 +++
.../translation/helpers/EncoderHelpers.java | 49 +-
.../translation/utils/ScalaInterop.java | 6 +-
.../translation/helpers/EncoderHelpersTest.java | 11 +-
runners/spark/job-server/spark_job_server.gradle | 7 +-
runners/spark/spark_runner.gradle | 158 +-
.../beam/runners/spark/SparkPipelineRunner.java | 2 +-
.../spark/coders/SparkRunnerKryoRegistrator.java | 28 +-
.../apache/beam/runners/spark/io/SourceRDD.java | 10 +-
.../runners/spark/io/SparkUnboundedSource.java | 2 +-
.../SparkGroupAlsoByWindowViaWindowSet.java | 10 +-
.../spark/stateful/SparkStateInternals.java | 2 +-
.../SparkStructuredStreamingPipelineOptions.java | 0
.../SparkStructuredStreamingPipelineResult.java | 3 +-
.../SparkStructuredStreamingRunner.java | 0
.../SparkStructuredStreamingRunnerRegistrar.java | 0
.../structuredstreaming/examples/WordCount.java | 0
.../io/BoundedDatasetFactory.java | 2 +-
.../spark/structuredstreaming/io/package-info.java | 0
.../structuredstreaming/metrics/BeamMetricSet.java | 0
.../metrics/MetricsAccumulator.java | 0
.../metrics/SparkBeamMetric.java | 0
.../metrics/SparkBeamMetricSource.java | 0
.../metrics/WithMetricsSupport.java | 0
.../structuredstreaming/metrics/package-info.java | 0
.../metrics/sink/CodahaleCsvSink.java | 0
.../metrics/sink/CodahaleGraphiteSink.java | 0
.../metrics/sink/package-info.java | 0
.../spark/structuredstreaming/package-info.java | 0
.../translation/EvaluationContext.java | 10 +-
.../translation/PipelineTranslator.java | 0
.../translation/SparkSessionFactory.java | 0
.../translation/SparkTransformOverrides.java | 0
.../translation/TransformTranslator.java | 0
.../translation/batch/Aggregators.java | 0
.../batch/CombineGloballyTranslatorBatch.java | 0
.../batch/CombineGroupedValuesTranslatorBatch.java | 0
.../batch/CombinePerKeyTranslatorBatch.java | 2 +-
.../batch/DoFnPartitionIteratorFactory.java | 0
.../translation/batch/DoFnRunnerFactory.java | 2 +-
.../translation/batch/DoFnRunnerWithMetrics.java | 0
.../translation/batch/FlattenTranslatorBatch.java | 0
.../translation/batch/GroupByKeyHelpers.java | 4 +-
.../batch/GroupByKeyTranslatorBatch.java | 2 +-
.../translation/batch/ImpulseTranslatorBatch.java | 0
.../translation/batch/ParDoTranslatorBatch.java | 0
.../translation/batch/PipelineTranslatorBatch.java | 0
.../batch/ReadSourceTranslatorBatch.java | 0
.../batch/ReshuffleTranslatorBatch.java | 0
.../batch/WindowAssignTranslatorBatch.java | 0
.../batch/functions/CachedSideInputReader.java | 0
.../GroupAlsoByWindowViaOutputBufferFn.java | 0
.../batch/functions/NoOpStepContext.java | 0
.../batch/functions/SideInputValues.java | 0
.../batch/functions/SparkSideInputReader.java | 0
.../translation/batch/functions/package-info.java | 0
.../translation/batch/package-info.java | 0
.../translation/helpers/CoderHelpers.java | 0
.../translation/helpers/EncoderFactory.java | 27 +-
.../translation/helpers/EncoderHelpers.java | 25 +-
.../translation/helpers/EncoderProvider.java | 0
.../translation/helpers/package-info.java | 0
.../translation/package-info.java | 0
.../translation/utils/ScalaInterop.java | 0
.../translation/utils/package-info.java | 0
.../spark/translation/GroupCombineFunctions.java | 2 +-
.../spark/translation/SparkContextFactory.java | 2 +-
.../SparkStreamingPortablePipelineTranslator.java | 3 +-
.../spark/translation/TransformTranslator.java | 4 +
.../spark/translation/TranslationUtils.java | 10 +-
.../translation/streaming/ParDoStateUpdateFn.java | 4 +-
.../streaming/StatefulStreamingParDoEvaluator.java | 4 +
.../streaming/StreamingTransformTranslator.java | 7 +-
.../apache/beam/runners/spark/util/ByteArray.java | 2 +-
.../runners/spark/util/SideInputBroadcast.java | 2 +-
.../beam/runners/spark/util/SideInputStorage.java | 2 +-
.../apache/beam/runners/spark/util/TimerUtils.java | 12 +
.../runners/spark/SparkRunnerRegistrarTest.java | 4 +-
.../aggregators/metrics/sink/InMemoryMetrics.java | 26 +-
.../coders/SparkRunnerKryoRegistratorTest.java | 52 +-
.../spark/metrics/SparkMetricsPusherTest.java | 2 +-
.../structuredstreaming/SparkSessionRule.java | 0
...parkStructuredStreamingRunnerRegistrarTest.java | 6 +-
.../StructuredStreamingPipelineStateTest.java | 0
.../aggregators/metrics/sink/InMemoryMetrics.java | 26 +-
.../metrics/sink/InMemoryMetricsSinkRule.java | 0
.../metrics/sink/SparkMetricsSinkTest.java | 0
.../metrics/SparkBeamMetricTest.java | 0
.../translation/EvaluationContextTest.java | 84 +
.../translation/batch/AggregatorsTest.java | 0
.../translation/batch/CombineGloballyTest.java | 0
.../batch/CombineGroupedValuesTest.java | 0
.../translation/batch/CombinePerKeyTest.java | 0
.../translation/batch/ComplexSourceTest.java | 0
.../translation/batch/FlattenTest.java | 0
.../translation/batch/GroupByKeyTest.java | 0
.../translation/batch/ParDoTest.java | 0
.../translation/batch/SimpleSourceTest.java | 0
.../translation/batch/WindowAssignTest.java | 0
.../batch/functions/SideInputValuesTest.java | 0
.../translation/helpers/EncoderHelpersTest.java | 2 +-
.../translation/streaming/SimpleSourceTest.java | 0
.../spark/translation/TransformTranslatorTest.java | 64 +
.../ValueAndCoderLazySerializableTest.java | 9 +-
scripts/beam-sql.sh | 7 +-
scripts/ci/issue-report/package-lock.json | 14 +-
scripts/ci/issue-report/package.json | 2 +-
scripts/ci/pr-bot/package-lock.json | 55 +-
scripts/ci/release/comment_pr_trigger_phrases.sh | 27 -
.../test/resources/jenkins_trigger_phrases.txt | 19 -
scripts/ci/release/test/resources/mass_comment.txt | 106 -
sdks/go.mod | 184 +-
sdks/go.sum | 1247 ++--------
sdks/go/container/Dockerfile | 10 +-
sdks/go/container/boot.go | 18 +-
sdks/go/container/boot_test.go | 14 +-
sdks/go/container/build.gradle | 6 +-
.../go/examples/large_wordcount/large_wordcount.go | 1 -
.../minimal_wordcount/minimal_wordcount.go | 23 +
.../ordered_list_state/ordered_list_state.go | 93 +
sdks/go/examples/snippets/04transforms.go | 32 +
sdks/go/pkg/beam/artifact/materialize.go | 45 +-
sdks/go/pkg/beam/artifact/materialize_test.go | 110 +
sdks/go/pkg/beam/artifact/options.go | 48 +
sdks/go/pkg/beam/artifact/options_test.go | 78 +
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/go/pkg/beam/core/graph/fn.go | 4 +-
.../core/runtime/contextreg/contextreg_test.go | 4 +-
sdks/go/pkg/beam/core/runtime/exec/data.go | 6 +
.../pkg/beam/core/runtime/exec/sideinput_test.go | 15 +
sdks/go/pkg/beam/core/runtime/exec/translate.go | 2 +
sdks/go/pkg/beam/core/runtime/exec/userstate.go | 150 ++
.../pkg/beam/core/runtime/exec/userstate_test.go | 25 +-
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 16 +-
sdks/go/pkg/beam/core/runtime/harness/statemgr.go | 82 +
sdks/go/pkg/beam/core/state/state.go | 133 ++
sdks/go/pkg/beam/core/state/state_test.go | 243 +-
sdks/go/pkg/beam/io/fhirio/common.go | 7 +-
sdks/go/pkg/beam/io/xlang/debeziumio/debezium.go | 40 +
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 182 +-
sdks/go/pkg/beam/runners/dataflow/dataflow_test.go | 38 +-
.../beam/runners/dataflow/dataflowlib/execute.go | 9 +-
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 93 +-
.../beam/runners/dataflow/dataflowlib/job_test.go | 79 +
.../prism/internal/engine/elementmanager.go | 4 +-
sdks/go/pkg/beam/runners/prism/internal/execute.go | 6 +-
.../beam/runners/prism/internal/execute_test.go | 43 +
sdks/go/pkg/beam/runners/prism/internal/stage.go | 17 +-
.../beam/runners/prism/internal/testdofns_test.go | 39 +
.../beam/runners/prism/internal/worker/bundle.go | 58 +-
.../runners/prism/internal/worker/worker_test.go | 67 +
sdks/go/pkg/beam/runners/samza/samza.go | 35 -
sdks/go/pkg/beam/transforms/stats/quantiles.go | 20 +-
sdks/go/pkg/beam/x/beamx/run.go | 1 -
sdks/go/run_with_go_version.sh | 19 +-
sdks/go/test/build.gradle | 39 +-
sdks/go/test/integration/expansions.go | 2 +-
sdks/go/test/integration/expansions_test.go | 1 -
sdks/go/test/integration/integration.go | 55 +-
.../integration/internal/containers/containers.go | 3 +-
.../integration/io/mongodbio/mongodbio_test.go | 1 -
.../test/integration/io/spannerio/test_helpers.go | 3 +-
.../integration/io/xlang/debezium/debezium_test.go | 1 -
.../go/test/integration/io/xlang/jdbc/jdbc_test.go | 8 +-
.../test/integration/io/xlang/kafka/kafka_test.go | 1 -
.../test/integration/primitives/primitives_test.go | 1 -
sdks/go/test/integration/primitives/state.go | 38 +
sdks/go/test/integration/primitives/state_test.go | 5 +
.../test/integration/synthetic/synthetic_test.go | 2 +-
.../test/integration/wordcount/wordcount_test.go | 1 -
sdks/go/test/integration/xlang/xlang_test.go | 1 -
sdks/go/test/regression/lperror_test.go | 1 -
sdks/go/test/regression/pardo_test.go | 1 -
sdks/go/test/run_validatesrunner_tests.sh | 38 +-
.../src/main/resources/beam/beam-codestyle.xml | 6 +-
sdks/java/container/Dockerfile | 31 +-
sdks/java/container/boot.go | 3 +
.../container/license_scripts/dep_urls_java.yaml | 12 +-
.../container/license_scripts/license_script.sh | 20 +-
sdks/java/core/build.gradle | 1 +
.../apache/beam/sdk/jmh/util/VarIntBenchmark.java | 2 +-
.../org/apache/beam/sdk/coders/AtomicCoder.java | 1 +
.../java/org/apache/beam/sdk/coders/Coder.java | 5 +
.../org/apache/beam/sdk/coders/DelegateCoder.java | 1 +
.../java/org/apache/beam/sdk/coders/RowCoder.java | 1 +
.../apache/beam/sdk/coders/RowCoderGenerator.java | 28 +-
.../apache/beam/sdk/coders/SerializableCoder.java | 1 +
.../beam/sdk/coders/StringDelegateCoder.java | 1 +
.../apache/beam/sdk/coders/StructuredCoder.java | 1 +
.../java/org/apache/beam/sdk/coders/ZstdCoder.java | 2 +-
.../sdk/fn/data/BeamFnDataOutboundAggregator.java | 13 +-
.../org/apache/beam/sdk/fn/data/WeightedList.java | 11 +-
.../apache/beam/sdk/io/DefaultFilenamePolicy.java | 11 +-
.../main/java/org/apache/beam/sdk/io/FileIO.java | 2 +-
.../java/org/apache/beam/sdk/io/FileSystem.java | 4 +-
.../java/org/apache/beam/sdk/io/FileSystems.java | 5 +-
.../java/org/apache/beam/sdk/io/TextSource.java | 2 +-
.../org/apache/beam/sdk/io/UnboundedSource.java | 1 +
.../org/apache/beam/sdk/io/range/ByteKeyRange.java | 4 +-
.../org/apache/beam/sdk/io/range/OffsetRange.java | 2 +-
.../org/apache/beam/sdk/lineage/LineageBase.java | 46 +
.../apache/beam/sdk/lineage/LineageOptions.java | 31 +-
.../org/apache/beam/sdk/lineage}/package-info.java | 14 +-
.../BoundedTrieMetricsLineage.java} | 36 +-
.../java/org/apache/beam/sdk/metrics/Lineage.java | 120 +-
.../beam/sdk/metrics/MetricsEnvironment.java | 11 +-
.../beam/sdk/metrics/StringSetMetricsLineage.java | 32 +-
.../options/DefaultPipelineOptionsRegistrar.java | 2 +
.../beam/sdk/options/PipelineOptionsFactory.java | 26 +-
.../beam/sdk/options/ProxyInvocationHandler.java | 1 +
.../apache/beam/sdk/options/SdkHarnessOptions.java | 6 +
.../GenerateSequenceSchemaTransformProvider.java | 12 +-
.../apache/beam/sdk/schemas/CachingFactory.java | 2 +-
.../beam/sdk/schemas/FromRowUsingCreator.java | 2 +-
.../sdk/schemas/GetterBasedSchemaProvider.java | 3 +-
.../apache/beam/sdk/schemas/JavaBeanSchema.java | 6 +-
.../java/org/apache/beam/sdk/schemas/Schema.java | 19 +-
.../org/apache/beam/sdk/schemas/SchemaCoder.java | 2 +-
.../apache/beam/sdk/schemas/SchemaTranslation.java | 2 +
.../io/payloads/JsonPayloadSerializerProvider.java | 53 +-
.../apache/beam/sdk/schemas/logicaltypes/Date.java | 20 +-
.../sdk/schemas/logicaltypes/EnumerationType.java | 2 +-
.../beam/sdk/schemas/logicaltypes/OneOfType.java | 2 +-
.../schemas/logicaltypes/UnknownLogicalType.java | 2 +-
.../apache/beam/sdk/schemas/transforms/Group.java | 4 +-
.../apache/beam/sdk/schemas/transforms/Select.java | 7 +-
.../providers/LoggingTransformProvider.java | 6 +-
.../beam/sdk/schemas/utils/ByteBuddyUtils.java | 130 +-
.../sdk/schemas/utils/StaticSchemaInference.java | 19 +-
.../apache/beam/sdk/state/WatermarkHoldState.java | 10 +
.../java/org/apache/beam/sdk/testing/PAssert.java | 2 +-
.../org/apache/beam/sdk/testing/StaticWindows.java | 3 +-
.../beam/sdk/testing/TestOutputReceiver.java | 2 +
.../beam/sdk/transforms/ApproximateQuantiles.java | 4 +-
.../beam/sdk/transforms/ApproximateUnique.java | 2 +-
.../apache/beam/sdk/transforms/BatchElements.java | 608 +++++
.../org/apache/beam/sdk/transforms/CombineFns.java | 2 +-
.../java/org/apache/beam/sdk/transforms/Count.java | 2 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 68 +-
.../beam/sdk/transforms/DoFnOutputReceivers.java | 26 +-
.../org/apache/beam/sdk/transforms/DoFnTester.java | 54 +-
.../org/apache/beam/sdk/transforms/Flatten.java | 2 +-
.../beam/sdk/transforms/GroupIntoBatches.java | 18 +-
.../org/apache/beam/sdk/transforms/JsonToRow.java | 6 +-
.../org/apache/beam/sdk/transforms/MapKeys.java | 1 -
.../beam/sdk/transforms/PeriodicSequence.java | 2 +-
.../apache/beam/sdk/transforms/Redistribute.java | 2 +
.../java/org/apache/beam/sdk/transforms/Reify.java | 15 +-
.../org/apache/beam/sdk/transforms/Reshuffle.java | 1 +
.../java/org/apache/beam/sdk/transforms/Sum.java | 6 +-
.../beam/sdk/transforms/join/RawUnionValue.java | 2 +-
.../reflect/ByteBuddyDoFnInvokerFactory.java | 55 +
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 74 +
.../beam/sdk/transforms/reflect/DoFnSignature.java | 135 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 37 +
.../transforms/resourcehints/ResourceHints.java | 2 +-
.../beam/sdk/util/ByteStringOutputStream.java | 12 +-
.../org/apache/beam/sdk/util/HistogramData.java | 2 +-
.../beam/sdk/util/RowJsonValueExtractors.java | 5 +-
.../beam/sdk/util/RowStringInterpolator.java | 8 +-
.../java/org/apache/beam/sdk/util/StreamUtils.java | 47 +-
.../util/UnboundedScheduledExecutorService.java | 12 +-
.../apache/beam/sdk/util/UnownedInputStream.java | 4 +
.../main/java/org/apache/beam/sdk/util/VarInt.java | 2 +-
.../org/apache/beam/sdk/util/WindowTracing.java | 3 +-
.../sdk/util/construction/PTransformMatchers.java | 2 +-
.../util/construction/PTransformTranslation.java | 7 +-
.../construction/SplittableParDoNaiveBounded.java | 58 +-
.../apache/beam/sdk/util/construction/Timer.java | 5 +-
.../sdk/util/construction/TransformUpgrader.java | 8 +-
.../sdk/util/construction/UnknownCoderWrapper.java | 2 +-
.../sdk/values/OpenTelemetryContextPropagator.java | 71 +
.../org/apache/beam/sdk/values/OutputBuilder.java | 5 +
.../apache/beam/sdk/values/PCollectionViews.java | 1 +
.../java/org/apache/beam/sdk/values/RowUtils.java | 2 +-
.../beam/sdk/values/ValueInSingleWindow.java | 87 +-
.../java/org/apache/beam/sdk/values/ValueKind.java | 29 +-
.../org/apache/beam/sdk/values/ValueKindUtil.java | 54 +
.../org/apache/beam/sdk/values/WindowedValue.java | 8 +
.../org/apache/beam/sdk/values/WindowedValues.java | 381 ++-
.../org/apache/beam/sdk/RequiresStableInputIT.java | 7 +-
.../apache/beam/sdk/coders/CoderRegistryTest.java | 4 +-
.../org/apache/beam/sdk/coders/RowCoderTest.java | 19 +
.../beam/sdk/coders/SerializableCoderTest.java | 7 +-
.../coders/TimestampPrefixingWindowCoderTest.java | 2 +-
.../fn/data/BeamFnDataOutboundAggregatorTest.java | 5 +-
.../apache/beam/sdk/io/GenerateSequenceTest.java | 5 +-
.../org/apache/beam/sdk/io/TextIOReadTest.java | 6 +-
.../org/apache/beam/sdk/io/TextIOWriteTest.java | 6 +-
.../apache/beam/sdk/lineage/LineagePluginTest.java | 272 +++
.../org/apache/beam/sdk/lineage/TestLineage.java | 77 +
.../sdk/options/PipelineOptionsFactoryTest.java | 5 +-
.../sdk/options/ProxyInvocationHandlerTest.java | 8 +-
.../beam/sdk/schemas/AutoValueSchemaTest.java | 4 +-
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 97 +-
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 136 +-
.../apache/beam/sdk/schemas/SchemaCoderTest.java | 10 +-
.../beam/sdk/schemas/SchemaTranslationTest.java | 3 +
.../sdk/schemas/logicaltypes/LogicalTypesTest.java | 4 +-
.../beam/sdk/schemas/transforms/CoGroupTest.java | 2 +-
.../beam/sdk/schemas/transforms/ConvertTest.java | 107 +-
.../beam/sdk/schemas/transforms/SelectTest.java | 8 +-
.../TypedSchemaTransformProviderTest.java | 4 +-
.../beam/sdk/schemas/utils/JavaBeanUtilsTest.java | 10 +
.../beam/sdk/schemas/utils/JsonUtilsTest.java | 8 +-
.../beam/sdk/schemas/utils/POJOUtilsTest.java | 32 +
.../beam/sdk/schemas/utils/TestJavaBeans.java | 130 +-
.../apache/beam/sdk/schemas/utils/TestPOJOs.java | 171 +-
.../apache/beam/sdk/testing/ExpectedLogsTest.java | 22 +-
.../apache/beam/sdk/testing/TestPipelineTest.java | 9 +-
.../beam/sdk/transforms/BatchElementsTest.java | 598 +++++
.../apache/beam/sdk/transforms/CombineFnsTest.java | 2 +-
.../apache/beam/sdk/transforms/DoFnTesterTest.java | 2 +-
.../sdk/transforms/MetadataPropagationTest.java | 167 ++
.../beam/sdk/transforms/ParDoSchemaTest.java | 8 +-
.../beam/sdk/transforms/SplittableDoFnTest.java | 46 +-
.../org/apache/beam/sdk/transforms/ToJsonTest.java | 6 +-
.../apache/beam/sdk/transforms/ValueKindTest.java | 623 +++++
.../org/apache/beam/sdk/transforms/ViewTest.java | 1 +
.../org/apache/beam/sdk/transforms/WaitTest.java | 2 +-
.../sdk/transforms/display/DisplayDataTest.java | 5 +-
.../beam/sdk/transforms/join/CoGbkResultTest.java | 2 +-
.../sdk/transforms/reflect/DefaultPackageTest.java | 2 +-
.../sdk/transforms/reflect/DoFnInvokersTest.java | 3 +
.../reflect/DoFnSignaturesSplittableDoFnTest.java | 12 +-
.../sdk/transforms/reflect/DoFnSignaturesTest.java | 110 +-
.../beam/sdk/transforms/windowing/WindowTest.java | 2 +-
.../beam/sdk/util/ByteStringOutputStreamTest.java | 8 +
.../org/apache/beam/sdk/util/CoderUtilsTest.java | 10 +
.../beam/sdk/util/SerializableUtilsTest.java | 3 +-
.../org/apache/beam/sdk/util/StreamUtilsTest.java | 95 +
.../UnboundedScheduledExecutorServiceTest.java | 2 +-
.../apache/beam/sdk/util/WindowedValueTest.java | 57 +-
.../util/construction/CombineTranslationTest.java | 2 +-
.../construction/PTransformReplacementsTest.java | 2 +-
.../construction/PTransformTranslationTest.java | 2 +-
.../util/construction/ParDoTranslationTest.java | 21 +-
.../util/construction/PipelineTranslationTest.java | 2 +-
.../sdk/util/construction/ReadTranslationTest.java | 4 +-
.../sdk/util/construction/SplittableParDoTest.java | 12 +-
.../beam/sdk/util/construction/TimerTest.java | 5 +-
.../construction/WindowIntoTranslationTest.java | 2 +-
.../WindowingStrategyTranslationTest.java | 4 +-
.../construction/graph/FieldAccessVisitorTest.java | 7 +-
.../graph/ProjectionProducerVisitorTest.java | 2 +-
.../graph/ProjectionPushdownOptimizerTest.java | 9 +-
.../construction/graph/QueryablePipelineTest.java | 2 +-
.../java/org/apache/beam/sdk/values/RowTest.java | 30 +-
.../sdk/expansion/service/ExpansionService.java | 14 +-
.../ExpansionServiceSchemaTransformProvider.java | 2 +-
.../expansion/service/ExpansionServiceTest.java | 2 +-
sdks/java/extensions/avro/build.gradle | 3 +-
.../beam/sdk/extensions/avro/coders/AvroCoder.java | 4 +-
.../sdk/extensions/avro/io/AvroDatumFactory.java | 2 +-
.../schemas/utils/AvroJavaTimeConversions.java | 4 +-
.../extensions/avro/schemas/utils/AvroUtils.java | 95 +-
.../avro/AvroVersionVerificationTest.java | 5 +-
.../sdk/extensions/avro/coders/AvroCoderTest.java | 9 +-
.../avro/io/AvroGeneratedUserFactory.java | 6 +-
.../sdk/extensions/avro/io/AvroSourceTest.java | 11 +-
.../avro/io/SerializableAvroCodecFactoryTest.java | 4 +-
.../extensions/avro/schemas/SchemaCoderTest.java | 2 +-
.../avro/schemas/TestAvroConversionFactory.java | 5 +-
.../extensions/avro/schemas/TestAvroFactory.java | 6 +-
.../avro/schemas/transforms/ConvertTest.java | 4 +-
.../avro/schemas/utils/AvroUtilsTest.java | 35 +-
.../extensions/avro/vendored-test/build.gradle | 38 -
.../sdk/extensions/gcp/options/GcpOptions.java | 19 +-
.../sdk/extensions/gcp/util/GceMetadataUtil.java | 1 +
.../beam/sdk/extensions/gcp/util/GcsUtil.java | 25 +-
.../beam/sdk/extensions/gcp/util/GcsUtilV1.java | 69 +-
.../beam/sdk/extensions/gcp/util/GcsUtilV2.java | 163 +-
.../gcp/util/RetryHttpRequestInitializer.java | 8 +-
.../sdk/extensions/gcp/util/gcsfs/GcsPath.java | 2 +-
.../beam/sdk/extensions/gcp/util/GcsUtilIT.java | 9 +-
.../gcp/util/GcsUtilParameterizedIT.java | 116 +-
.../beam/sdk/extensions/gcp/util/GcsUtilTest.java | 19 +-
.../extensions/jackson/JacksonTransformsTest.java | 2 +-
.../apache/beam/sdk/extensions/kryo/KryoCoder.java | 2 +-
.../beam/sdk/extensions/kryo/KryoCoderTest.java | 4 +-
.../beam/sdk/extensions/ml/DLPDeidentifyText.java | 4 +-
.../beam/sdk/extensions/ml/DLPInspectText.java | 4 +-
.../beam/sdk/extensions/ml/DLPReidentifyText.java | 4 +-
.../ml/RecommendationAIImportCatalogItems.java | 6 +-
.../ml/RecommendationAIImportUserEvents.java | 6 +-
.../ordered/GlobalSequencesProcessorDoFn.java | 14 +-
.../ordered/OrderedProcessingHandler.java | 2 +-
.../ordered/OrderedProcessingStatus.java | 2 +-
.../beam/sdk/extensions/ordered/ProcessorDoFn.java | 11 +-
.../ordered/SequencePerKeyProcessorDoFn.java | 4 +-
.../ordered/combiner/DefaultSequenceCombiner.java | 2 +-
.../ordered/OrderedEventProcessorTestBase.java | 2 +-
.../sdk/extensions/protobuf/DynamicProtoCoder.java | 1 +
.../beam/sdk/extensions/protobuf/ProtoCoder.java | 1 +
.../beam/sdk/extensions/python/PythonService.java | 6 +-
.../sdk/extensions/sbe/SbeLogicalTypesTest.java | 7 +-
.../sorter/BufferedExternalSorterTest.java | 4 +-
sdks/java/extensions/sql/iceberg/build.gradle | 1 +
.../sdk/extensions/sql/jdbc/BeamSqlLineIT.java | 1 +
.../sdk/extensions/sql/example/model/Customer.java | 2 +-
.../sdk/extensions/sql/example/model/Order.java | 2 +-
.../sdk/extensions/sql/impl/CatalogSchema.java | 2 +-
.../beam/sdk/extensions/sql/impl/nfa/NFA.java | 31 +-
.../extensions/sql/impl/parser/SqlAlterTable.java | 2 +-
.../sql/impl/parser/SqlCreateExternalTable.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamCalcRel.java | 143 +-
.../sql/impl/rel/BeamEnumerableConverter.java | 4 +-
.../sdk/extensions/sql/impl/rel/BeamJoinRel.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamWindowRel.java | 10 +-
.../sql/impl/rule/BeamSideInputLookupJoinRule.java | 7 +-
.../sql/impl/transform/agg/CovarianceFn.java | 2 +-
.../sql/impl/transform/agg/VarianceFn.java | 2 +-
.../extensions/sql/impl/utils/CalciteUtils.java | 33 +
.../sql/meta/catalog/CatalogManager.java | 4 +-
.../provider/bigquery/BeamSqlUnparseContext.java | 2 +-
.../sql/meta/provider/bigquery/BigQueryTable.java | 2 +-
.../sql/meta/provider/kafka/BeamKafkaTable.java | 2 +-
.../sql/meta/provider/text/TextTable.java | 2 +-
.../sdk/extensions/sql/BeamComplexTypeTest.java | 28 +-
.../extensions/sql/InferredJavaBeanSqlTest.java | 4 +-
.../apache/beam/sdk/extensions/sql/TestUtils.java | 2 +-
.../sql/impl/rel/BeamEnumerableConverterTest.java | 2 +-
.../sql/impl/schema/BeamSqlRowCoderTest.java | 7 +-
.../sql/impl/utils/CalciteUtilsTest.java | 20 +
.../meta/provider/mongodb/MongoDbReadWriteIT.java | 44 +-
.../beam/sdk/extensions/timeseries/FillGaps.java | 8 +-
.../zetasketch/ApproximateCountDistinct.java | 8 +-
.../BigQueryHllSketchCompatibilityIT.java | 7 +-
.../fn/harness/jmh/ProcessBundleBenchmark.java | 6 +-
.../java/org/apache/beam/fn/harness/Caches.java | 57 +-
.../org/apache/beam/fn/harness/CombineRunners.java | 4 +-
.../beam/fn/harness/ExternalWorkerService.java | 2 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 201 +-
.../java/org/apache/beam/fn/harness/FnHarness.java | 35 +-
.../beam/fn/harness/PrecombineGroupingTable.java | 4 +-
.../SplittablePairWithRestrictionDoFnRunner.java | 6 +
...littableSplitAndSizeRestrictionsDoFnRunner.java | 2 +-
.../fn/harness/control/ExecutionStateSampler.java | 37 +-
.../fn/harness/control/ProcessBundleHandler.java | 2 +-
.../apache/beam/fn/harness/debug/DataSampler.java | 3 +-
.../fn/harness/logging/BeamFnLoggingClient.java | 3 +-
.../apache/beam/fn/harness/state/BagUserState.java | 2 +-
.../beam/fn/harness/state/FnApiStateAccessor.java | 2 +-
.../fn/harness/state/StateFetchingIterators.java | 195 +-
.../beam/fn/harness/status/BeamFnStatusClient.java | 1 +
.../beam/fn/harness/WindowMergingFnRunnerTest.java | 8 +-
.../harness/control/ProcessBundleHandlerTest.java | 2 +-
.../state/LazyCachingIteratorToIterableTest.java | 10 +-
.../harness/state/StateFetchingIteratorsTest.java | 39 +
.../sdk/io/aws2/common/AsyncBatchWriteHandler.java | 6 +-
.../apache/beam/sdk/io/aws2/common/ObjectPool.java | 2 +-
.../sdk/io/aws2/kinesis/EFOShardSubscriber.java | 2 +-
.../apache/beam/sdk/io/aws2/kinesis/KinesisIO.java | 4 +-
.../beam/sdk/io/aws2/kinesis/ShardCheckpoint.java | 2 +-
.../beam/sdk/io/aws2/kinesis/StartingPoint.java | 2 +-
.../sdk/io/aws2/schemas/AwsSchemaProvider.java | 2 +-
.../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 2 +-
.../beam/sdk/io/aws2/sqs/SqsCheckpointMark.java | 2 +-
.../io/aws2/common/AsyncBatchWriteHandlerTest.java | 1 +
.../sdk/io/aws2/dynamodb/DynamoDBIOWriteTest.java | 2 +-
.../sdk/io/aws2/dynamodb/testing/DynamoDBIOIT.java | 2 +-
.../aws2/kinesis/EFOShardSubscribersPoolTest.java | 2 +-
.../sdk/io/aws2/kinesis/KinesisIOWriteTest.java | 10 +-
.../sdk/io/aws2/options/SerializationTestUtil.java | 2 +-
sdks/java/io/amqp/build.gradle | 4 +-
.../org/apache/beam/sdk/io/amqp/AmqpIOTest.java | 2 +-
.../azure/blobstore/AzureBlobStoreFileSystem.java | 1 +
.../blobstore/AzureBlobStoreFileSystemTest.java | 3 +-
.../beam/sdk/io/cassandra/CassandraIOTest.java | 8 +-
.../org/apache/beam/sdk/io/cdap/ConfigWrapper.java | 2 +-
.../io/cdap/PluginConfigInstantiationUtils.java | 4 +-
.../java/org/apache/beam/sdk/io/cdap/CdapIOIT.java | 2 +-
.../apache/beam/sdk/io/cdap/TestRowDBWritable.java | 6 +-
sdks/java/io/clickhouse/build.gradle | 2 +
.../beam/sdk/io/clickhouse/ClickHouseIO.java | 41 +-
.../{ClickHouseIOTest.java => ClickHouseIOIT.java} | 50 +-
.../beam/sdk/io/clickhouse/ClickHouseIOTest.java | 469 +---
.../beam/sdk/io/common/DatabaseTestHelper.java | 11 +-
.../org/apache/beam/sdk/io/common/HashingFn.java | 2 +-
.../beam/sdk/io/common/SchemaAwareJavaBeans.java | 36 +-
.../org/apache/beam/sdk/io/common/TestRow.java | 10 +-
.../beam/sdk/io/datadog/DatadogEventPublisher.java | 10 +-
.../org/apache/beam/sdk/io/datadog/DatadogIO.java | 2 -
.../org/apache/beam/io/debezium/DebeziumIO.java | 76 +
.../DebeziumReadSchemaTransformProvider.java | 4 +-
.../io/debezium/DebeziumTransformRegistrar.java | 38 +
.../beam/io/debezium/FileSystemOffsetRetainer.java | 166 ++
.../apache/beam/io/debezium/KafkaConnectUtils.java | 27 +-
.../beam/io/debezium/KafkaSourceConsumerFn.java | 26 +-
.../apache/beam/io/debezium/OffsetRetainer.java | 69 +
.../io/debezium/DebeziumIOMySqlConnectorIT.java | 2 +-
.../apache/beam/io/debezium/DebeziumIOTest.java | 153 ++
.../io/debezium/FileSystemOffsetRetainerTest.java | 131 +
.../beam/io/debezium/KafkaConnectSchemaTest.java | 41 +-
.../java/io/delta/build.gradle | 32 +-
.../java/org/apache/beam/sdk/io/delta/DeltaIO.java | 91 +
.../apache/beam/sdk/io/delta}/package-info.java | 8 +-
.../org/apache/beam/sdk/io/delta/DeltaIOTest.java | 62 +
.../io/elasticsearch/ElasticsearchIOTestUtils.java | 3 +-
.../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 18 +-
sdks/java/io/expansion-service/build.gradle | 33 +-
.../beam/sdk/io/common/FileBasedIOITHelper.java | 3 +-
.../sdk/io/fileschematransform/XmlRowAdapter.java | 2 +-
.../sdk/io/fileschematransform/XmlRowValue.java | 2 +-
.../AvroReadSchemaTransformFormatProviderTest.java | 11 +
.../FileReadSchemaTransformFormatProviderTest.java | 2 +-
...WriteSchemaTransformFormatProviderTestData.java | 61 +
.../JsonReadSchemaTransformFormatProviderTest.java | 8 +-
...rquetReadSchemaTransformFormatProviderTest.java | 11 +
.../io/fileschematransform/XmlRowValueTest.java | 14 +-
.../sdk/io/googleads/DummyRateLimitPolicy.java | 2 +-
.../expansion-service/build.gradle | 6 -
.../sdk/io/gcp/bigquery/AppendClientCache.java | 140 ++
.../beam/sdk/io/gcp/bigquery/AppendClientInfo.java | 78 +-
.../AvroGenericRecordToStorageApiProto.java | 6 +-
.../io/gcp/bigquery/BeamRowToStorageApiProto.java | 4 +-
.../sdk/io/gcp/bigquery/BigQueryAvroUtils.java | 2 +-
.../bigquery/BigQueryCoderProviderRegistrar.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 87 +-
.../sdk/io/gcp/bigquery/BigQueryIOTranslation.java | 47 +-
.../sdk/io/gcp/bigquery/BigQueryInsertError.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryOptions.java | 15 +
.../beam/sdk/io/gcp/bigquery/BigQueryServices.java | 6 +-
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 56 +-
.../sdk/io/gcp/bigquery/BigQuerySinkMetrics.java | 4 +-
.../io/gcp/bigquery/BigQueryStorageSourceBase.java | 71 +-
.../gcp/bigquery/BigQueryStorageStreamSource.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 4 +-
.../sdk/io/gcp/bigquery/ConvertMessagesDoFn.java | 297 +++
.../gcp/bigquery/DynamicDestinationsHelpers.java | 2 +-
.../sdk/io/gcp/bigquery/MergeSchemaCombineFn.java | 76 +
.../io/gcp/bigquery/PassThroughThenCleanup.java | 3 +-
.../sdk/io/gcp/bigquery/PatchTableSchemaDoFn.java | 163 ++
.../beam/sdk/io/gcp/bigquery/RetryManager.java | 8 +
.../io/gcp/bigquery/RowMutationInformation.java | 16 +-
.../sdk/io/gcp/bigquery/SchemaUpdateHoldingFn.java | 259 ++
.../sdk/io/gcp/bigquery/SplittingIterable.java | 76 +-
.../io/gcp/bigquery/StorageApiConvertMessages.java | 307 ++-
.../bigquery/StorageApiDynamicDestinations.java | 15 +-
.../StorageApiDynamicDestinationsBeamRow.java | 16 +-
...StorageApiDynamicDestinationsGenericRecord.java | 17 +-
.../StorageApiDynamicDestinationsProto.java | 16 +-
.../StorageApiDynamicDestinationsTableRow.java | 139 +-
.../gcp/bigquery/StorageApiFinalizeWritesDoFn.java | 16 +-
.../bigquery/StorageApiFlushAndFinalizeDoFn.java | 13 +-
.../beam/sdk/io/gcp/bigquery/StorageApiLoads.java | 23 +-
.../io/gcp/bigquery/StorageApiWritePayload.java | 9 +
.../bigquery/StorageApiWriteUnshardedRecords.java | 290 +--
.../bigquery/StorageApiWritesShardedRecords.java | 1097 +++++----
.../io/gcp/bigquery/TableRowToStorageApiProto.java | 468 +++-
.../beam/sdk/io/gcp/bigquery/TableSchemaCache.java | 121 +-
.../bigquery/TwoLevelMessageConverterCache.java | 9 +-
.../io/gcp/bigquery/UpdateSchemaDestination.java | 11 +-
.../sdk/io/gcp/bigquery/UpgradeTableSchema.java | 308 +++
.../beam/sdk/io/gcp/bigquery/WritePartition.java | 6 +-
.../beam/sdk/io/gcp/bigquery/WriteTables.java | 8 +-
...ueryStorageWriteApiSchemaTransformProvider.java | 4 +-
.../io/gcp/bigtable/BigtableConfigTranslator.java | 2 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 6 +-
.../io/gcp/bigtable/BigtableServiceFactory.java | 14 +-
.../BigtableWriteSchemaTransformProvider.java | 2 +-
.../action/DetectNewPartitionsAction.java | 8 +-
.../gcp/bigtable/changestreams/dao/DaoFactory.java | 1 +
.../changestreams/dao/MetadataTableDao.java | 4 +-
.../changestreams/dofn/InitializeDoFn.java | 6 +-
.../beam/sdk/io/gcp/datastore/EntityToRow.java | 2 +-
.../beam/sdk/io/gcp/datastore/RowToEntity.java | 2 +-
.../sdk/io/gcp/firestore/FirestoreV1WriteFn.java | 2 +-
.../apache/beam/sdk/io/gcp/healthcare/FhirIO.java | 39 +-
.../io/gcp/healthcare/FhirIOPatientEverything.java | 7 +-
.../sdk/io/gcp/healthcare/FhirSearchParameter.java | 2 +-
.../apache/beam/sdk/io/gcp/healthcare/HL7v2IO.java | 34 +-
.../io/gcp/healthcare/HttpHealthcareApiClient.java | 14 +-
.../sdk/io/gcp/pubsub/AddTimestampAttribute.java | 7 +-
.../beam/sdk/io/gcp/pubsub/PubsubClient.java | 8 +-
.../beam/sdk/io/gcp/pubsub/PubsubMessages.java | 52 +-
.../pubsub/PubsubReadSchemaTransformProvider.java | 2 +-
.../beam/sdk/io/gcp/pubsub/PubsubRowToMessage.java | 8 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSource.java | 2 +-
.../apache/beam/sdk/io/gcp/pubsub/TestPubsub.java | 1 +
.../beam/sdk/io/gcp/pubsub/TestPubsubSignal.java | 1 +
.../beam/sdk/io/gcp/spanner/BatchSpannerRead.java | 4 +-
.../beam/sdk/io/gcp/spanner/MutationUtils.java | 4 +-
.../beam/sdk/io/gcp/spanner/NaiveSpannerRead.java | 4 +-
.../beam/sdk/io/gcp/spanner/ReadSpannerSchema.java | 2 +-
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 14 +
.../beam/sdk/io/gcp/spanner/SpannerConfig.java | 25 +
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 172 +-
.../SpannerWriteSchemaTransformProvider.java | 4 +-
.../changestreams/ChangeStreamsConstants.java | 23 +
.../changestreams/action/ActionFactory.java | 12 +-
.../action/ChildPartitionsRecordAction.java | 8 +-
.../action/DetectNewPartitionsAction.java | 9 +-
.../action/HeartbeatRecordAction.java | 14 +-
.../action/PartitionStartRecordAction.java | 6 +-
.../action/QueryChangeStreamAction.java | 58 +-
.../spanner/changestreams/dao/ChangeStreamDao.java | 50 +-
.../gcp/spanner/changestreams/dao/DaoFactory.java | 12 +
.../changestreams/dao/PartitionMetadataDao.java | 186 +-
.../spanner/changestreams/dofn/InitializeDoFn.java | 93 +-
.../dofn/ReadChangeStreamPartitionDoFn.java | 23 +-
.../mapper/ChangeStreamRecordMapper.java | 1 -
.../mapper/PartitionMetadataMapper.java | 5 +-
.../changestreams/model/PartitionMetadata.java | 32 +-
.../beam/sdk/io/gcp/testing/BigqueryClient.java | 33 +-
.../beam/sdk/io/gcp/testing/BigqueryMatcher.java | 4 +-
.../sdk/io/gcp/testing/FakeBigQueryServices.java | 3 +-
.../sdk/io/gcp/testing/FakeDatasetService.java | 238 +-
.../cloud/spanner/FakeBatchTransactionId.java | 2 +-
.../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java | 1 +
.../gcp/bigquery/BeamRowToStorageApiProtoTest.java | 11 +-
.../sdk/io/gcp/bigquery/BigQueryIOReadTest.java | 2 +-
.../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 110 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 205 +-
.../gcp/bigquery/BigQueryTableRowEqualityTest.java | 202 ++
.../StorageApiDataTriggeredSchemaUpdateIT.java | 372 +++
.../beam/sdk/io/gcp/bigquery/TableRowMatchers.java | 204 ++
.../bigquery/TableRowToStorageApiProtoTest.java | 344 ++-
.../io/gcp/bigquery/UpgradeTableSchemaTest.java | 361 +++
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 16 +-
.../beam/sdk/io/gcp/bigtable/BigtableReadIT.java | 6 +-
.../BigtableReadSchemaTransformProviderIT.java | 7 +-
.../io/gcp/bigtable/BigtableServiceImplTest.java | 2 -
...gtableSimpleWriteSchemaTransformProviderIT.java | 6 +-
.../beam/sdk/io/gcp/bigtable/BigtableWriteIT.java | 5 +-
.../BigtableWriteSchemaTransformProviderIT.java | 6 +-
.../ReadChangeStreamPartitionActionTest.java | 10 +-
.../changestreams/dao/MetadataTableDaoTest.java | 2 +-
.../changestreams/it/BigtableChangeStreamIT.java | 2 +-
.../beam/sdk/io/gcp/datastore/DatastoreV1Test.java | 1 +
.../beam/sdk/io/gcp/pubsub/PubsubClientTest.java | 3 +-
.../beam/sdk/io/gcp/pubsub/PubsubMessagesTest.java | 176 ++
.../sdk/io/gcp/pubsub/PubsubUnboundedSinkTest.java | 2 +-
.../gcp/spanner/SpannerIOReadChangeStreamTest.java | 61 +
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 18 +-
.../beam/sdk/io/gcp/spanner/SpannerReadIT.java | 5 +
.../beam/sdk/io/gcp/spanner/SpannerWriteIT.java | 5 +
.../SpannerChangeStreamErrorTest.java | 51 +-
.../action/ChildPartitionsRecordActionTest.java | 6 +-
.../action/HeartbeatRecordActionTest.java | 149 +-
.../action/PartitionStartRecordActionTest.java | 39 +-
.../action/QueryChangeStreamActionTest.java | 185 +-
.../changestreams/dao/ChangeStreamDaoTest.java | 65 +-
.../dao/ChangeStreamResultSetTest.java | 1 -
.../dao/PartitionMetadataDaoTest.java | 117 +-
.../changestreams/dofn/InitializeDoFnTest.java | 80 +-
.../dofn/ReadChangeStreamPartitionDoFnTest.java | 19 +-
.../changestreams/it/IntegrationTestEnv.java | 60 +-
...StreamOrderedByTimestampAndTransactionIdIT.java | 86 +-
...nnerChangeStreamOrderedWithinKeyGloballyIT.java | 16 +-
.../it/SpannerChangeStreamOrderedWithinKeyIT.java | 22 +-
.../it/SpannerChangeStreamPlacementTableIT.java | 53 +-
...pannerChangeStreamPlacementTablePostgresIT.java | 38 +-
.../it/SpannerChangeStreamPostgresIT.java | 3 +-
...SpannerChangeStreamTransactionBoundariesIT.java | 18 +-
.../mapper/PartitionMetadataMapperTest.java | 11 +-
.../changestreams/model/ModelEncodingTest.java | 3 +-
.../changestreams/model/PartitionMetadataTest.java | 56 +-
.../spanner/changestreams/util/TestJsonMapper.java | 2 +-
.../beam/sdk/io/gcp/storage/GcsKmsKeyIT.java | 7 +-
.../apache/beam/sdk/io/gcp/storage/GcsMatchIT.java | 17 +-
.../apache/beam/sdk/io/hdfs/HadoopFileSystem.java | 17 +
.../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 20 +
.../beam/sdk/io/hadoop/format/HadoopFormatIO.java | 39 +-
.../apache/beam/sdk/io/hadoop/format/Employee.java | 2 +-
.../sdk/io/hadoop/format/TestRowDBWritable.java | 6 +-
sdks/java/io/hbase/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/hbase/HBaseIO.java | 14 +-
.../beam/sdk/io/hbase/HBaseRowMutationsCoder.java | 2 -
.../beam/sdk/io/hbase/HBaseSharedConnection.java | 2 +-
.../org/apache/beam/sdk/io/hbase/HBaseIOIT.java | 2 +-
.../org/apache/beam/sdk/io/hbase/HBaseIOTest.java | 2 +-
.../sdk/io/hcatalog/HiveDatabaseTestHelper.java | 17 +-
sdks/java/io/iceberg/build.gradle | 16 +-
sdks/java/io/iceberg/hive/build.gradle | 2 +-
.../org/apache/beam/sdk/io/iceberg/AddFiles.java | 901 +++++++
.../iceberg/AddFilesSchemaTransformProvider.java | 199 ++
.../beam/sdk/io/iceberg/AppendFilesToTables.java | 18 +-
.../iceberg/AssignDestinationsAndPartitions.java | 181 ++
.../apache/beam/sdk/io/iceberg/BeamRowWrapper.java | 180 ++
.../apache/beam/sdk/io/iceberg/FilterUtils.java | 1 +
.../beam/sdk/io/iceberg/IcebergCatalogConfig.java | 46 +-
.../org/apache/beam/sdk/io/iceberg/IcebergIO.java | 73 +-
.../beam/sdk/io/iceberg/IcebergScanConfig.java | 22 +-
.../sdk/io/iceberg/IcebergTableCreateConfig.java | 14 +
.../apache/beam/sdk/io/iceberg/IcebergUtils.java | 25 +-
.../IcebergWriteSchemaTransformProvider.java | 41 +
.../apache/beam/sdk/io/iceberg/PartitionUtils.java | 5 +-
.../io/iceberg/PortableIcebergDestinations.java | 4 +
.../org/apache/beam/sdk/io/iceberg/ReadUtils.java | 30 +
.../apache/beam/sdk/io/iceberg/RecordWriter.java | 3 +-
.../beam/sdk/io/iceberg/RecordWriterManager.java | 30 +-
.../apache/beam/sdk/io/iceberg/ScanTaskReader.java | 5 +-
.../beam/sdk/io/iceberg/SerializableDataFile.java | 16 +-
.../apache/beam/sdk/io/iceberg/SortOrderUtils.java | 135 ++
.../sdk/io/iceberg/WriteDirectRowsToFiles.java | 2 +-
.../io/iceberg/WritePartitionedRowsToFiles.java | 263 ++
.../beam/sdk/io/iceberg/WriteToPartitions.java | 118 +
.../org/apache/beam/sdk/io/iceberg/AddFilesIT.java | 538 +++++
.../apache/beam/sdk/io/iceberg/AddFilesTest.java | 709 ++++++
.../beam/sdk/io/iceberg/BeamRowWrapperTest.java | 226 ++
.../beam/sdk/io/iceberg/IcebergIOReadTest.java | 30 +
.../beam/sdk/io/iceberg/IcebergIOWriteTest.java | 390 ++-
.../IcebergWriteSchemaTransformProviderTest.java | 50 +-
.../sdk/io/iceberg/RecordWriterManagerTest.java | 80 +-
.../sdk/io/iceberg/SerializableDataFileTest.java | 65 +
.../beam/sdk/io/iceberg/SortOrderUtilsTest.java | 125 +
.../io/iceberg/catalog/IcebergCatalogBaseIT.java | 53 +
.../io/iceberg/catalog/hiveutils/ScriptRunner.java | 2 +-
.../apache/beam/sdk/io/influxdb/GenerateData.java | 10 +-
.../beam/sdk/io/influxdb/InfluxDbIOTest.java | 1 +
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 6 +-
.../io/jdbc/JdbcReadSchemaTransformProvider.java | 4 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java | 19 +-
.../io/jdbc/JdbcWriteSchemaTransformProvider.java | 6 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java | 5 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 3 +-
.../beam/sdk/io/jdbc/JdbcSchemaIOProviderTest.java | 3 +-
.../apache/beam/sdk/io/jdbc/JdbcTestHelper.java | 2 +-
.../org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java | 7 +-
sdks/java/io/jms/build.gradle | 2 +-
.../apache/beam/sdk/io/jms/JmsCheckpointMark.java | 2 +-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 2 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOTest.java | 10 +-
.../jms/MockNonSerializableConnectionFactory.java | 21 +
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 144 +-
.../beam/sdk/io/kafka/KafkaSourceDescriptor.java | 4 +-
.../beam/sdk/io/kafka/KafkaUnboundedReader.java | 7 +-
.../org/apache/beam/sdk/io/kafka/KafkaWriter.java | 2 +-
.../beam/sdk/io/kafka/ReadFromKafkaDoFn.java | 4 +-
.../beam/sdk/io/kafka/KafkaIOExternalTest.java | 86 +
.../org/apache/beam/sdk/io/kafka/KafkaIOIT.java | 5 +-
.../KafkaUnboundedReaderIdlePartitionTest.java | 2 +-
.../java/org/apache/beam/sdk/io/kudu/KuduIO.java | 6 +-
.../org/apache/beam/sdk/io/kudu/KuduIOTest.java | 4 +
sdks/java/io/mongodb/build.gradle | 2 +-
.../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 2 +-
.../org/apache/beam/sdk/io/mongodb/MongoDbIO.java | 8 +-
.../beam/sdk/io/mongodb/MongoDBGridFSIOTest.java | 66 +-
.../apache/beam/sdk/io/mongodb/MongoDBIOIT.java | 3 +-
.../apache/beam/sdk/io/mongodb/MongoDbIOTest.java | 73 +-
.../org/apache/beam/sdk/io/mqtt/MqttIOTest.java | 8 +-
.../java/org/apache/beam/sdk/io/neo4j/Neo4jIO.java | 19 +-
.../org/apache/beam/sdk/io/parquet/ParquetIO.java | 40 +-
.../apache/beam/sdk/io/pulsar/PulsarIOUtils.java | 3 -
.../org/apache/beam/sdk/io/pulsar/PulsarIOIT.java | 2 +-
.../apache/beam/sdk/io/pulsar/PulsarIOTest.java | 9 +-
.../beam/sdk/io/rabbitmq/RabbitMqMessage.java | 7 +-
.../beam/sdk/io/rabbitmq/ExchangeTestPlan.java | 2 +-
sdks/java/io/redis/build.gradle | 2 +-
.../org/apache/beam/sdk/io/redis/RedisCursor.java | 2 +-
.../org/apache/beam/sdk/io/redis/RedisIOTest.java | 60 +-
.../org/apache/beam/io/requestresponse/Cache.java | 4 +
.../org/apache/beam/io/requestresponse/Call.java | 108 +-
.../apache/beam/io/requestresponse/Monitoring.java | 170 +-
.../apache/beam/io/requestresponse/Repeater.java | 1 +
.../apache/beam/io/requestresponse/CallTest.java | 4 +-
.../EchoGRPCCallerWithSetupTeardown.java | 1 +
.../io/requestresponse/RequestResponseIOTest.java | 8 +-
.../beam/sdk/io/singlestore/SingleStoreIO.java | 9 +-
.../apache/beam/sdk/io/singlestore/TestHelper.java | 2 +-
sdks/java/io/snowflake/build.gradle | 8 +-
.../apache/beam/sdk/io/snowflake/SnowflakeIO.java | 64 +-
.../test/FakeSnowflakeBasicDataSource.java | 6 +-
.../test/FakeSnowflakeBatchServiceImpl.java | 2 +-
.../io/snowflake/test/FakeSnowflakeDatabase.java | 12 +-
.../test/unit/DataSourceConfigurationTest.java | 2 +-
.../test/unit/write/SchemaDispositionTest.java | 2 +-
.../test/unit/write/SnowflakeIOWriteTest.java | 2 +-
.../test/unit/write/StreamingWriteTest.java | 2 +-
.../beam/sdk/io/solace/broker/SessionService.java | 14 +-
.../org/apache/beam/sdk/io/solace/data/Solace.java | 4 +-
.../sdk/io/solace/read/SolaceCheckpointMark.java | 6 +-
...entialsBasicAuthJcsmpSessionServiceFactory.java | 2 +-
.../sdk/io/solace/it/SolaceContainerManager.java | 2 +-
.../java/org/apache/beam/sdk/io/solr/SolrIO.java | 1 +
.../sparkreceiver/RabbitMqReceiverWithOffset.java | 4 +-
.../sdk/io/sparkreceiver/SparkReceiverIOIT.java | 7 +-
.../beam/sdk/io/splunk/HttpEventPublisher.java | 4 +-
.../beam/sdk/io/synthetic/SyntheticStep.java | 2 +
.../org/apache/beam/sdk/io/thrift/ThriftIO.java | 6 +-
.../testing/TestSchemaTransformProvider.java | 6 +-
.../META-INF/maven/archetype-metadata.xml | 2 +-
.../src/main/resources/archetype-resources/pom.xml | 18 +-
.../resources/projects/basic/archetype.properties | 2 +-
.../src/main/resources/archetype-resources/pom.xml | 17 +-
.../src/main/java/StarterPipeline.java | 19 +-
.../resources/projects/basic/archetype.properties | 2 +-
.../ml/inference/openai/OpenAIModelHandlerIT.java | 2 +-
.../ml/inference/remote/RemoteInferenceTest.java | 18 +-
.../java/org/apache/beam/sdk/jpmstests/JpmsIT.java | 6 +-
.../beam/sdk/testing/TestPipelineExtension.java | 1 -
.../beam/sdk/nexmark/NexmarkConfiguration.java | 5 +-
.../org/apache/beam/sdk/nexmark/NexmarkUtils.java | 2 +-
.../org/apache/beam/sdk/nexmark/model/Auction.java | 2 +-
.../apache/beam/sdk/nexmark/model/AuctionBid.java | 2 +-
.../beam/sdk/nexmark/model/AuctionCount.java | 2 +-
.../beam/sdk/nexmark/model/AuctionPrice.java | 2 +-
.../org/apache/beam/sdk/nexmark/model/Bid.java | 2 +-
.../beam/sdk/nexmark/model/BidsPerSession.java | 2 +-
.../beam/sdk/nexmark/model/CategoryPrice.java | 2 +-
.../org/apache/beam/sdk/nexmark/model/Done.java | 2 +-
.../org/apache/beam/sdk/nexmark/model/Event.java | 2 +-
.../beam/sdk/nexmark/model/IdNameReserve.java | 2 +-
.../beam/sdk/nexmark/model/NameCityStateId.java | 2 +-
.../org/apache/beam/sdk/nexmark/model/Person.java | 2 +-
.../apache/beam/sdk/nexmark/model/SellerPrice.java | 2 +-
.../apache/beam/sdk/nexmark/queries/Query10.java | 25 +-
.../apache/beam/sdk/nexmark/queries/Query5.java | 2 +-
.../beam/sdk/nexmark/queries/WinningBids.java | 4 +-
.../sdk/nexmark/sources/generator/Generator.java | 2 +-
.../nexmark/sources/generator/GeneratorConfig.java | 2 +-
sdks/java/testing/test-utils/build.gradle | 2 +-
.../testutils/jvmverification/JvmVerification.java | 17 +-
.../sdk/testutils/metrics/MetricsReaderTest.java | 2 +-
.../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java | 1 +
.../apache/beam/sdk/tpcds/SqlTransformRunner.java | 1 +
.../beam/sdk/tpcds/TpcdsParametersReader.java | 2 -
.../org/apache/beam/sdk/tpcds/TpcdsRunResult.java | 2 +
.../sdk/testing/watermarks/WatermarkLatency.java | 2 +-
.../launcher/TransformServiceLauncher.java | 12 +-
sdks/python/.isort.cfg | 58 -
sdks/python/apache_beam/__init__.py | 2 +-
sdks/python/apache_beam/coders/coder_impl.pxd | 4 +
sdks/python/apache_beam/coders/coder_impl.py | 27 +-
sdks/python/apache_beam/coders/coders.py | 31 +-
.../apache_beam/coders/coders_test_common.py | 14 +-
sdks/python/apache_beam/coders/observable_test.py | 3 +-
sdks/python/apache_beam/coders/row_coder.py | 5 +-
sdks/python/apache_beam/coders/row_coder_test.py | 8 +-
sdks/python/apache_beam/coders/slow_stream.py | 3 +-
.../apache_beam/coders/standard_coders_test.py | 4 +-
sdks/python/apache_beam/coders/typecoders.py | 54 +-
sdks/python/apache_beam/dataframe/frame_base.py | 3 +-
sdks/python/apache_beam/dataframe/io.py | 20 +-
sdks/python/apache_beam/dataframe/schemas_test.py | 85 +-
.../apache_beam/examples/inference/README.md | 74 +-
.../composite_transform.py | 3 +-
.../inference/pytorch_language_modeling.py | 3 +-
.../examples/inference/pytorch_sentiment.py | 80 +-
.../examples/inference/rate_limiter_vertex_ai.py | 2 +
.../inference/sklearn_examples_requirements.txt | 3 +-
.../examples/inference/table_row_batch_example.py | 194 ++
.../examples/inference/table_row_inference.py | 369 +++
.../examples/inference/table_row_inference_test.py | 196 ++
.../inference/table_row_inference_utils.py | 310 +++
.../inference/tfx_bsl/build_tensorflow_model.py | 3 +-
.../examples/inference/vllm_gemma_batch.py | 1 -
.../examples/inference/vllm_text_completion.py | 53 +-
.../apache_beam/examples/matrix_power_test.py | 13 +-
.../kfp/components/preprocessing/requirements.txt | 2 +-
.../apache_beam/examples/rate_limiter_simple.py | 4 +
.../apache_beam/examples/snippets/snippets.py | 124 +
.../apache_beam/examples/snippets/snippets_test.py | 13 +-
.../transforms/aggregation/groupby_test.py | 1 -
.../apache_beam/internal/cloudpickle_pickler.py | 13 +-
sdks/python/apache_beam/internal/dill_pickler.py | 6 +-
sdks/python/apache_beam/internal/metrics/metric.py | 12 +-
sdks/python/apache_beam/internal/util.py | 10 +-
sdks/python/apache_beam/io/avroio.py | 6 +-
sdks/python/apache_beam/io/avroio_test.py | 4 +-
.../apache_beam/io/components/rate_limiter.py | 4 +-
sdks/python/apache_beam/io/debezium.py | 45 +-
.../io/external/xlang_jdbcio_it_test.py | 15 +-
.../io/external/xlang_kafkaio_it_test.py | 2 +-
.../io/external/xlang_kafkaio_perf_test.py | 3 +-
.../python/apache_beam/io/filebasedio_perf_test.py | 3 +-
sdks/python/apache_beam/io/filebasedsource.py | 3 +-
sdks/python/apache_beam/io/fileio.py | 21 +-
sdks/python/apache_beam/io/filesystem.py | 8 +-
.../io/flink/flink_streaming_impulse_source.py | 3 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 39 +-
.../apache_beam/io/gcp/bigquery_avro_tools.py | 9 +-
.../apache_beam/io/gcp/bigquery_change_history.py | 1346 +++++++++++
.../io/gcp/bigquery_change_history_it_test.py | 638 +++++
.../io/gcp/bigquery_change_history_test.py | 209 ++
.../apache_beam/io/gcp/bigquery_file_loads.py | 8 +-
.../apache_beam/io/gcp/bigquery_read_internal.py | 6 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 13 +-
sdks/python/apache_beam/io/gcp/bigtableio.py | 4 +-
.../apache_beam/io/gcp/datastore/v1new/helper.py | 3 +-
.../apache_beam/io/gcp/datastore/v1new/types.py | 3 +-
.../apache_beam/io/gcp/experimental/spannerio.py | 8 +-
sdks/python/apache_beam/io/gcp/pubsub.py | 6 +-
sdks/python/apache_beam/io/iobase.py | 5 +-
sdks/python/apache_beam/io/jdbc.py | 8 +-
sdks/python/apache_beam/io/kafka.py | 31 +-
sdks/python/apache_beam/io/requestresponse.py | 19 +-
.../apache_beam/io/requestresponse_it_test.py | 3 +-
sdks/python/apache_beam/io/restriction_trackers.py | 4 +-
sdks/python/apache_beam/io/textio.py | 3 +-
sdks/python/apache_beam/metrics/cells.py | 1 +
sdks/python/apache_beam/metrics/execution.py | 1 +
sdks/python/apache_beam/metrics/metric.py | 41 +-
sdks/python/apache_beam/metrics/metricbase.py | 3 +-
.../python/apache_beam/metrics/monitoring_infos.py | 6 +-
sdks/python/apache_beam/ml/gcp/cloud_dlp.py | 3 +-
.../python/apache_beam/ml/gcp/naturallanguageml.py | 5 +-
.../apache_beam/ml/gcp/recommendations_ai.py | 33 +-
.../apache_beam/ml/gcp/recommendations_ai_test.py | 8 +-
.../ml/gcp/recommendations_ai_test_it.py | 7 +-
.../apache_beam/ml/gcp/videointelligenceml.py | 3 +-
sdks/python/apache_beam/ml/gcp/visionml.py | 10 +-
.../ml/inference/agent_development_kit.py | 308 +++
.../ml/inference/agent_development_kit_test.py | 344 +++
.../ml/inference/anthropic_inference.py | 296 +++
.../ml/inference/anthropic_inference_it_test.py | 215 ++
.../ml/inference/anthropic_inference_test.py | 351 +++
.../inference/anthropic_tests_requirements.txt} | 4 +-
sdks/python/apache_beam/ml/inference/base_test.py | 45 +-
.../apache_beam/ml/inference/gemini_inference.py | 10 +
.../ml/inference/huggingface_inference.py | 24 +
.../apache_beam/ml/inference/model_manager.py | 8 +-
.../apache_beam/ml/inference/onnx_inference.py | 8 +
.../apache_beam/ml/inference/pytorch_inference.py | 32 +-
.../apache_beam/ml/inference/sklearn_inference.py | 16 +
...ts.txt => table_row_inference_requirements.txt} | 10 +-
.../ml/inference/tensorflow_inference.py | 16 +
.../ml/inference/tensorflow_tests_requirements.txt | 3 +
.../apache_beam/ml/inference/tensorrt_inference.py | 8 +
.../ml/inference/vertex_ai_inference.py | 10 +
.../apache_beam/ml/inference/vllm_inference.py | 29 +-
.../ml/inference/vllm_tests_requirements.txt | 3 +-
.../apache_beam/ml/inference/xgboost_inference.py | 8 +
sdks/python/apache_beam/ml/rag/chunking/base.py | 5 +-
.../apache_beam/ml/rag/chunking/base_test.py | 3 +-
.../apache_beam/ml/rag/chunking/langchain.py | 8 +-
sdks/python/apache_beam/ml/rag/embeddings/base.py | 5 +-
.../ml/rag/embeddings/vertex_ai_test.py | 9 +-
.../ml/rag/enrichment/bigquery_vector_search.py | 21 +-
.../apache_beam/ml/rag/enrichment/milvus_search.py | 49 +-
.../ml/rag/enrichment/milvus_search_it_test.py | 12 +-
.../python/apache_beam/ml/rag/ingestion/alloydb.py | 12 +-
.../ml/rag/ingestion/alloydb_it_test.py | 5 +-
.../apache_beam/ml/rag/ingestion/bigquery.py | 7 +-
.../apache_beam/ml/rag/ingestion/cloudsql.py | 22 +-
.../ml/rag/ingestion/cloudsql_it_test.py | 3 +-
.../apache_beam/ml/rag/ingestion/jdbc_common.py | 8 +-
.../apache_beam/ml/rag/ingestion/milvus_search.py | 12 +-
.../ml/rag/ingestion/milvus_search_it_test.py | 12 +-
sdks/python/apache_beam/ml/rag/ingestion/mysql.py | 15 +-
.../apache_beam/ml/rag/ingestion/mysql_common.py | 25 +-
.../apache_beam/ml/rag/ingestion/postgres.py | 8 +-
.../ml/rag/ingestion/postgres_common.py | 30 +-
.../ml/rag/ingestion/postgres_it_test.py | 5 +-
.../python/apache_beam/ml/rag/ingestion/spanner.py | 26 +-
.../ml/rag/ingestion/spanner_it_test.py | 2 +-
.../apache_beam/ml/rag/ingestion/test_utils.py | 3 +-
sdks/python/apache_beam/ml/rag/test_utils.py | 63 +-
sdks/python/apache_beam/ml/rag/types.py | 17 +-
sdks/python/apache_beam/ml/rag/utils.py | 14 +-
sdks/python/apache_beam/ml/transforms/base.py | 22 +-
.../python/apache_beam/options/pipeline_options.py | 38 +-
.../apache_beam/options/pipeline_options_test.py | 6 +
sdks/python/apache_beam/options/value_provider.py | 3 +-
sdks/python/apache_beam/pipeline.py | 5 +-
sdks/python/apache_beam/portability/common_urns.py | 1 +
sdks/python/apache_beam/pvalue.py | 3 +-
sdks/python/apache_beam/runners/common.py | 1 +
.../python/apache_beam/runners/dask/dask_runner.py | 4 +-
sdks/python/apache_beam/runners/dask/overrides.py | 16 +-
.../runners/dask/transform_evaluator.py | 8 +-
.../runners/dataflow/internal/apiclient.py | 27 +-
.../runners/dataflow/internal/apiclient_test.py | 96 +-
.../clients/dataflow/dataflow_v1b3_messages.py | 104 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../apache_beam/runners/direct/bundle_factory.py | 5 +-
.../direct/consumer_tracking_pipeline_visitor.py | 9 +-
.../apache_beam/runners/direct/direct_runner.py | 20 +-
.../runners/direct/evaluation_context.py | 24 +-
sdks/python/apache_beam/runners/direct/executor.py | 11 +-
.../runners/direct/transform_evaluator.py | 14 +-
.../runners/direct/watermark_manager.py | 18 +-
.../apache_beam/runners/interactive/README.md | 15 +-
.../runners/interactive/augmented_pipeline.py | 6 +-
.../runners/interactive/cache_manager.py | 22 +-
.../runners/interactive/caching/read_cache.py | 6 +-
.../runners/interactive/caching/write_cache.py | 4 +-
.../dataproc/dataproc_cluster_manager.py | 5 +-
.../runners/interactive/display/pipeline_graph.py | 11 +-
.../interactive/display/pipeline_graph_renderer.py | 3 +-
.../apache-beam-jupyterlab-sidepanel/README.md | 52 +-
.../yaml_parse_utils.py | 12 +-
.../apache-beam-jupyterlab-sidepanel/install.json | 5 +
.../apache-beam-jupyterlab-sidepanel/package.json | 3 +-
.../pyproject.toml | 2 +
.../style/index.js | 13 +
.../apache-beam-jupyterlab-sidepanel/tsconfig.json | 1 +
.../runners/interactive/interactive_environment.py | 10 +-
.../runners/interactive/options/capture_control.py | 5 +-
.../runners/interactive/pipeline_instrument.py | 5 +-
.../runners/interactive/sql/beam_sql_magics.py | 21 +-
.../runners/interactive/sql/sql_chain.py | 10 +-
.../apache_beam/runners/interactive/sql/utils.py | 12 +-
.../runners/interactive/sql/utils_test.py | 3 +-
.../runners/interactive/testing/mock_env.py | 3 +-
.../runners/interactive/user_pipeline_tracker.py | 109 +-
.../interactive/user_pipeline_tracker_test.py | 48 +
.../apache_beam/runners/interactive/utils.py | 13 +-
.../python/apache_beam/runners/pipeline_context.py | 17 +-
.../runners/portability/abstract_job_service.py | 8 +-
.../runners/portability/artifact_service.py | 11 +-
.../runners/portability/expansion_service_main.py | 14 +-
.../runners/portability/expansion_service_test.py | 2 +-
.../runners/portability/fn_api_runner/execution.py | 79 +-
.../runners/portability/fn_api_runner/fn_runner.py | 61 +-
.../portability/fn_api_runner/fn_runner_test.py | 14 +-
.../portability/fn_api_runner/translations.py | 26 +-
.../portability/fn_api_runner/trigger_manager.py | 24 +-
.../fn_api_runner/visualization_tools.py | 6 +-
.../portability/fn_api_runner/watermark_manager.py | 19 +-
.../portability/fn_api_runner/worker_handlers.py | 6 +-
.../apache_beam/runners/portability/job_server.py | 14 +-
.../runners/portability/local_job_service.py | 9 +-
.../runners/portability/portable_runner.py | 17 +-
.../runners/portability/portable_runner_test.py | 117 +-
.../runners/portability/prism_runner.py | 15 +-
.../runners/portability/prism_runner_test.py | 73 +
.../runners/portability/samza_runner_test.py | 200 --
.../runners/portability/sdk_container_builder.py | 3 +-
.../apache_beam/runners/portability/stager.py | 32 +-
.../apache_beam/runners/portability/stager_test.py | 3 +-
sdks/python/apache_beam/runners/sdf_utils.py | 3 +-
sdks/python/apache_beam/runners/trivial_runner.py | 5 +-
.../apache_beam/runners/worker/bundle_processor.py | 135 +-
.../apache_beam/runners/worker/data_plane.py | 70 +-
.../apache_beam/runners/worker/data_sampler.py | 25 +-
.../runners/worker/data_sampler_test.py | 5 +-
.../apache_beam/runners/worker/log_handler.py | 3 +-
sdks/python/apache_beam/runners/worker/logger.py | 1 +
.../apache_beam/runners/worker/opcounters.py | 1 +
.../apache_beam/runners/worker/operations.py | 6 +
.../apache_beam/runners/worker/sdk_worker.py | 19 +-
.../apache_beam/runners/worker/sdk_worker_test.py | 32 +
.../apache_beam/runners/worker/statecache.py | 8 +-
.../apache_beam/runners/worker/statesampler.py | 3 +-
.../apache_beam/runners/worker/worker_pool_main.py | 17 +-
.../testing/analyzers/github_issues_utils.py | 12 +-
.../apache_beam/testing/analyzers/perf_analysis.py | 7 +-
.../testing/analyzers/perf_analysis_utils.py | 25 +-
.../testing/benchmarks/cloudml/requirements.txt | 12 +-
.../testing/benchmarks/inference/README.md | 15 +
.../inference/table_row_inference_benchmark.py | 113 +
.../testing/benchmarks/nexmark/queries/query3.py | 8 +-
.../testing/benchmarks/sort_and_batch_benchmark.py | 655 +++++
.../testing/load_tests/dataflow_cost_benchmark.py | 156 +-
.../apache_beam/testing/load_tests/load_test.py | 10 +-
.../testing/load_tests/load_test_metrics_utils.py | 12 +-
.../testing/load_tests/sideinput_test.py | 8 +-
.../apache_beam/testing/synthetic_pipeline.py | 3 +-
sdks/python/apache_beam/testing/test_pipeline.py | 18 +-
sdks/python/apache_beam/testing/util.py | 3 +-
.../tools/runtime_type_check_microbenchmark.py | 9 +-
sdks/python/apache_beam/transforms/async_dofn.py | 119 +-
.../apache_beam/transforms/async_dofn_test.py | 84 +-
.../apache_beam/transforms/combiners_test.py | 73 +-
sdks/python/apache_beam/transforms/core.py | 132 +-
sdks/python/apache_beam/transforms/core_test.py | 212 ++
sdks/python/apache_beam/transforms/cy_combiners.py | 1 +
sdks/python/apache_beam/transforms/deduplicate.py | 4 +-
.../transforms/enrichment_handlers/bigquery.py | 35 +-
.../enrichment_handlers/bigquery_test.py | 82 +
.../transforms/enrichment_handlers/cloudsql.py | 33 +-
.../enrichment_handlers/cloudsql_test.py | 53 +
.../python/apache_beam/transforms/external_test.py | 17 +-
.../external_transform_provider_it_test.py | 18 +-
.../transforms/managed_iceberg_it_test.py | 23 +-
.../apache_beam/transforms/ptransform_test.py | 98 +-
sdks/python/apache_beam/transforms/sideinputs.py | 3 +-
.../apache_beam/transforms/sideinputs_test.py | 43 +-
sdks/python/apache_beam/transforms/sql_test.py | 26 +
sdks/python/apache_beam/transforms/stats.py | 1 +
sdks/python/apache_beam/transforms/util.py | 316 ++-
sdks/python/apache_beam/transforms/util_test.py | 335 +++
.../transforms/validate_runner_xlang_test.py | 15 +-
.../typehints/arrow_type_compatibility.py | 1 +
sdks/python/apache_beam/typehints/batch_test.py | 1 +
sdks/python/apache_beam/typehints/decorators.py | 1 +
.../apache_beam/typehints/decorators_test.py | 1 +
.../typehints/native_type_compatibility.py | 55 +-
.../typehints/native_type_compatibility_test.py | 64 +
.../typehints/pandas_type_compatibility.py | 1 +
sdks/python/apache_beam/typehints/row_type.py | 51 +-
sdks/python/apache_beam/typehints/row_type_test.py | 177 ++
.../apache_beam/typehints/schema_registry.py | 11 +-
sdks/python/apache_beam/typehints/schemas.py | 135 +-
sdks/python/apache_beam/typehints/schemas_test.py | 86 +-
.../python/apache_beam/typehints/typecheck_test.py | 1 +
.../apache_beam/typehints/typed_pipeline_test.py | 1 +
sdks/python/apache_beam/typehints/typehints.py | 13 +-
.../python/apache_beam/typehints/typehints_test.py | 31 +-
.../byte_limited_queue.pxd} | 22 +-
.../python/apache_beam/utils/byte_limited_queue.py | 204 ++
.../apache_beam/utils/byte_limited_queue_test.py | 270 +++
sdks/python/apache_beam/utils/counters.py | 1 +
.../apache_beam/utils/multi_process_shared.py | 20 +-
.../apache_beam/utils/multi_process_shared_test.py | 63 +-
sdks/python/apache_beam/utils/proto_utils.py | 9 +-
sdks/python/apache_beam/utils/subprocess_server.py | 120 +-
.../apache_beam/utils/subprocess_server_test.py | 162 ++
sdks/python/apache_beam/utils/urns.py | 13 +-
sdks/python/apache_beam/utils/windowed_value.py | 3 +-
sdks/python/apache_beam/version.py | 2 +-
.../yaml/examples/testing/examples_test.py | 54 +-
.../databases/bigquery_dynamic_destinations.yaml | 97 +
.../yaml/extended_tests/messaging/kafka.yaml | 54 +-
sdks/python/apache_beam/yaml/integration_tests.py | 24 +-
sdks/python/apache_beam/yaml/readme_test.py | 2 +-
sdks/python/apache_beam/yaml/standard_io.yaml | 47 +-
.../apache_beam/yaml/tests/iceberg_add_files.yaml | 115 +
.../yaml/tests/iceberg_add_files_batch.yaml | 88 +
sdks/python/apache_beam/yaml/tests/reshuffle.yaml | 48 +
sdks/python/apache_beam/yaml/yaml_io.py | 15 +-
sdks/python/apache_beam/yaml/yaml_ml.py | 7 +-
sdks/python/apache_beam/yaml/yaml_provider.py | 28 +-
sdks/python/apache_beam/yaml/yaml_testing.py | 7 +-
sdks/python/apache_beam/yaml/yaml_transform.py | 88 +-
.../python/apache_beam/yaml/yaml_transform_test.py | 44 +-
sdks/python/apache_beam/yaml/yaml_utils.py | 3 +-
sdks/python/build.gradle | 5 +-
sdks/python/conftest.py | 2 +-
sdks/python/container/Dockerfile | 6 +-
sdks/python/container/boot.go | 33 +-
sdks/python/container/build.gradle | 3 +-
sdks/python/container/common.gradle | 25 +-
.../python/container/distroless/py314/build.gradle | 16 +-
.../container/license_scripts/dep_urls_py.yaml | 4 +-
.../container/license_scripts/pull_licenses_py.py | 2 +-
.../container/ml/py310/base_image_requirements.txt | 100 +-
.../container/ml/py310/gpu_image_requirements.txt | 26 +-
.../container/ml/py311/base_image_requirements.txt | 105 +-
.../container/ml/py311/gpu_image_requirements.txt | 31 +-
.../container/ml/py312/base_image_requirements.txt | 107 +-
.../container/ml/py312/gpu_image_requirements.txt | 39 +-
.../container/ml/py313/base_image_requirements.txt | 90 +-
.../python/container/ml/py314/build.gradle | 16 +-
.../container/py310/base_image_requirements.txt | 84 +-
.../container/py311/base_image_requirements.txt | 89 +-
.../container/py312/base_image_requirements.txt | 89 +-
.../container/py313/base_image_requirements.txt | 84 +-
.../{py313 => py314}/base_image_requirements.txt | 88 +-
.../python/container/py314/build.gradle | 16 +-
sdks/python/container/run_generate_requirements.sh | 4 +-
sdks/python/gen_managed_doc.py | 3 +-
sdks/python/gen_xlang_wrappers.py | 20 +-
sdks/python/mypy.ini | 110 -
sdks/python/pyproject.toml | 175 ++
sdks/python/pyrefly.toml | 62 -
sdks/python/pytest.ini | 1 +
sdks/python/scripts/{run_pylint.sh => run_lint.sh} | 13 +-
sdks/python/setup.py | 125 +-
sdks/python/test-suites/dataflow/common.gradle | 28 +
.../python/test-suites/dataflow/py314/build.gradle | 12 +-
.../python/test-suites/direct/py314/build.gradle | 12 +-
sdks/python/test-suites/gradle.properties | 29 +-
sdks/python/test-suites/portable/build.gradle | 6 -
sdks/python/test-suites/portable/common.gradle | 27 -
.../python/test-suites/portable/py314/build.gradle | 14 +-
.../tox/{pycommon => py314}/build.gradle | 23 +-
sdks/python/test-suites/tox/pycommon/build.gradle | 3 -
sdks/python/tox.ini | 60 +-
sdks/typescript/container/boot.go | 3 +
sdks/typescript/package-lock.json | 2516 ++++++++++++--------
sdks/typescript/package.json | 14 +-
sdks/typescript/test/primitives_test.ts | 12 +-
settings.gradle.kts | 30 +-
website/Dockerfile | 2 +-
website/www/site/assets/js/language-switch-v2.js | 2 +
.../www/site/assets/scss/_syntax-highlighting.scss | 3 +
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.71.0.md | 8 +-
website/www/site/content/en/blog/beam-2.72.0.md | 63 +
website/www/site/content/en/blog/beam-2.73.0.md | 67 +
.../www/site/content/en/blog/capability-matrix.md | 2 +-
website/www/site/content/en/blog/looping-timers.md | 11 +-
.../en/contribute/postcommits-policies-details.md | 2 +-
.../en/documentation/io/built-in/iceberg.md | 433 ++++
.../site/content/en/documentation/io/connectors.md | 164 +-
.../site/content/en/documentation/io/managed-io.md | 726 +++---
.../content/en/documentation/patterns/overview.md | 3 +
.../en/documentation/patterns/rate-limiting.md | 97 +
.../content/en/documentation/programming-guide.md | 87 +
.../site/content/en/documentation/runners/flink.md | 10 +
.../site/content/en/documentation/runners/samza.md | 4 +-
.../www/site/content/en/documentation/sdks/java.md | 2 +-
.../content/en/documentation/sdks/yaml-schema.md | 48 +
.../transforms/java/aggregation/batchelements.md | 31 +
.../en/documentation/transforms/java/overview.md | 1 +
.../site/content/en/get-started/beam-overview.md | 5 +-
.../www/site/content/en/get-started/downloads.md | 24 +-
.../site/content/en/get-started/quickstart-java.md | 21 -
.../content/en/get-started/wordcount-example.md | 43 -
website/www/site/content/en/performance/_index.md | 2 +
.../en/performance/tablerowinference/_index.md | 45 +
.../tablerowinferencestreaming/_index.md | 43 +
website/www/site/content/en/roadmap/_index.md | 2 +-
website/www/site/content/en/roadmap/go-sdk.md | 2 +-
.../www/site/content/en/roadmap/samza-runner.md | 2 +-
website/www/site/data/capability_matrix.yaml | 311 +--
website/www/site/data/performance.yaml | 32 +
.../partials/section-menu/en/documentation.html | 19 +-
.../layouts/partials/section-menu/en/roadmap.html | 1 -
.../layouts/partials/section-menu/en/runners.html | 1 -
.../www/site/layouts/shortcodes/button-colab.html | 2 +-
.../site/layouts/shortcodes/language-switcher.html | 3 +
.../shortcodes/{button-colab.html => section.html} | 14 +-
.../shortcodes/{button-colab.html => tab.html} | 26 +-
website/www/site/static/images/colab/icon32.png | Bin 0 -> 1418 bytes
website/www/yarn.lock | 19 +-
2114 files changed, 47737 insertions(+), 38688 deletions(-)
create mode 100644 .agent/skills/adding-new-metadata/SKILL.md
create mode 100644 .agent/skills/beam-dofn-modernizer/SKILL.md
create mode 100644 .agent/skills/yaml-development/SKILL.md
copy .github/trigger_files/{beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.json => beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_Engine.json} (100%)
delete mode 100644 .github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Samza.json
copy .github/trigger_files/{beam_PostCommit_Yaml_Xlang_Direct.json => beam_PostCommit_Python_Versions.json} (84%)
copy .github/trigger_files/{beam_PreCommit_Yaml_Xlang_Direct.json => beam_PreCommit_Python_PVR_Flink.json} (100%)
delete mode 100644 .github/workflows/beam_PostCommit_Go_VR_Samza.yml
rename .github/workflows/{beam_PostCommit_Java_PVR_Samza.yml => beam_PostCommit_Java_PVR_Spark4_Batch.yml} (71%)
copy .github/workflows/{beam_PostCommit_Java_PVR_Spark3_Streaming.yml => beam_PostCommit_Java_PVR_Spark4_Streaming.yml} (76%)
copy .github/workflows/{beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.yml => beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_Engine.yml} (80%)
delete mode 100644 .github/workflows/beam_PostCommit_Java_ValidatesRunner_Flink_Java8.yml
delete mode 100644 .github/workflows/beam_PostCommit_Java_ValidatesRunner_Samza.yml
copy .github/workflows/{beam_PostCommit_Java_ValidatesRunner_Spark.yml => beam_PostCommit_Java_ValidatesRunner_Spark4.yml} (78%)
delete mode 100644 .github/workflows/beam_PostCommit_Java_ValidatesRunner_Spark_Java8.yml
delete mode 100644 .github/workflows/beam_PostCommit_Python_ValidatesRunner_Samza.yml
copy .github/workflows/{beam_PreCommit_Python.yml => beam_PostCommit_Python_Versions.yml} (77%)
delete mode 100644 .github/workflows/beam_PostCommit_XVR_Samza.yml
copy .github/workflows/{beam_PreCommit_SQL.yml => beam_PreCommit_Java_Dataflow.yml} (54%)
rename .github/workflows/{beam_PreCommit_Java_Spark3_Versions.yml => beam_PreCommit_Java_Spark_Versions.yml} (76%)
delete mode 100644 .github/workflows/beam_PreCommit_SQL_Java8.yml
copy .github/workflows/load-tests-pipeline-options/{beam_Inference_Python_Benchmarks_Dataflow_VLLM_Gemma_Batch.txt => beam_Inference_Python_Benchmarks_Dataflow_Table_Row_Inference_Batch.txt} (59%)
copy .github/workflows/load-tests-pipeline-options/{beam_Inference_Python_Benchmarks_Dataflow_VLLM_Gemma_Batch.txt => beam_Inference_Python_Benchmarks_Dataflow_Table_Row_Inference_Stream.txt} (60%)
delete mode 100644 .test-infra/BUILD_STATUS.md
create mode 100644 examples/java/iceberg/src/main/java/org/apache/beam/examples/iceberg/snippets/IcebergBeamSchemaAndRow.java
create mode 100644 examples/java/iceberg/src/main/java/org/apache/beam/examples/iceberg/snippets/Quickstart.java
copy examples/java/src/main/java/org/apache/beam/examples/{TopExample.java => BatchElementsExample.java} (67%)
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/PicoWordCount.java
copy examples/java/src/main/java/org/apache/beam/examples/{RateLimiterSimple.java => RateLimitedPubSubReader.java} (69%)
delete mode 100644 release/src/main/scripts/jenkins_jobs.txt
delete mode 100644 release/src/main/scripts/mass_comment.py
create mode 100644 runners/core-java/src/main/java/org/apache/beam/runners/core/CombinedMetadata.java
create mode 100644 runners/core-java/src/main/java/org/apache/beam/runners/core/CombinedMetadataCombiner.java
rename runners/{samza/src/main/java/org/apache/beam/runners/samza/package-info.java => core-java/src/main/java/org/apache/beam/runners/core/MetadataCombiner.java} (81%)
create mode 100644 runners/core-java/src/main/java/org/apache/beam/runners/core/serialization/SentinelBitSetCoder.java
create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/CombinedMetadataTest.java
create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/serialization/SentinelBitSetCoderTest.java
create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/triggers/TriggerStateMachineRunnerTest.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillValueKindHelper.java
rename runners/{samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineExceptionContext.java => google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/SimpleByteHistogram.java} (52%)
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/FailoverChannel.java
copy sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/AvroVersionVerificationTest.java => runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/util/SimpleByteHistogramTest.java (53%)
delete mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDispatcherClientTest.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/FailoverChannelTest.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/NoopClientCall.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/work/processing/StreamingCommitFinalizerTest.java
delete mode 100644 runners/samza/build.gradle
delete mode 100644 runners/samza/job-server/build.gradle
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaExecutionContext.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaExecutionEnvironment.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaJobInvoker.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaJobServerDriver.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineLifeCycleListener.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptions.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineOptionsValidator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineResult.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPipelineRunner.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPortablePipelineOptions.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPortablePipelineResult.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunner.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunnerOverrideConfigs.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaRunnerRegistrar.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/TestSamzaRunner.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystem.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/UnboundedSourceSystem.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/container/BeamContainerRunner.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/container/BeamJobCoordinatorRunner.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/container/ContainerCfgLoader.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/container/ContainerCfgLoaderFactory.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/metrics/DoFnRunnerWithMetrics.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/metrics/FnWithMetricsWrapper.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/metrics/SamzaGBKMetricOp.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/metrics/SamzaMetricOp.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/metrics/SamzaMetricOpFactory.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/metrics/SamzaMetricsContainer.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/metrics/SamzaTransformMetricRegistry.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/metrics/SamzaTransformMetrics.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/AsyncDoFnRunner.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/BundleManager.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/ClassicBundleManager.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/DoFnRunnerWithKeyedInternals.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/FutureCollector.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/FutureCollectorImpl.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/GroupByKeyOp.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/KeyedInternals.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/KeyedTimerData.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/KvToKeyedWorkItemOp.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/Op.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OpAdapter.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OpMessage.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OutputManagerFactory.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/PortableBundleManager.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/PortableDoFnOp.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaAssignContext.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnRunners.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaExecutableStageContextFactory.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaMetricsBundleProgressHandler.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaStateRequestHandlers.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaStoreStateInternals.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactory.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SingletonKeyedWorkItem.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SplittableParDoProcessKeyedElementsOp.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/WindowAssignOp.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/package-info.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/state/SamzaMapState.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/state/SamzaSetState.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/state/package-info.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/transforms/GroupWithoutRepartition.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/transforms/package-info.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ConfigBuilder.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ConfigContext.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/FlattenPCollectionsTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/GroupByKeyTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ImpulseTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PViewToIdMapper.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ParDoBoundMultiTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/PortableTranslationContext.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ReadTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/RedistributeByKeyTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/ReshuffleTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaImpulseSystemFactory.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPipelineTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortablePipelineTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPortableTranslatorRegistrar.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishView.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTransformOverride.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaPublishViewTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaTestStreamSystemFactory.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaTestStreamTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaTransformOverrides.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SplittableParDoTranslators.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/StateIdParser.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TransformConfigGenerator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TransformTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/TranslationContext.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/WindowAssignTranslator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/package-info.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/ConfigUtils.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/DoFnUtils.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/FutureUtils.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/HashIdGenerator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/PipelineJsonRenderer.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/PortableConfigUtils.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/SamzaCoders.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/SamzaPipelineExceptionListener.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/SamzaPipelineTranslatorUtils.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/StateUtils.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/StoreIdGenerator.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/WindowUtils.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/package-info.java
delete mode 100644 runners/samza/src/main/resources/log4j.properties
delete mode 100644 runners/samza/src/main/resources/samza-conf.properties
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/SamzaPipelineOptionsValidatorTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/adapter/BoundedSourceSystemTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/adapter/TestBoundedSource.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/adapter/TestCheckpointMark.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/adapter/TestSourceHelpers.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/adapter/TestUnboundedSource.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/adapter/UnboundedSourceSystemTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/metrics/TestSamzaRunnerWithTransformMetrics.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/metrics/TestSamzaTransformMetricsRegistry.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/AsyncDoFnRunnerTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/ClassicBundleManagerTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/FutureCollectorImplTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/GroupByKeyOpTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/KeyedTimerDataTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/PortableBundleManagerTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaMetricsBundleProgressHandlerTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaStoreStateInternalsTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SamzaTimerInternalsFactoryTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/SdkHarnessDoFnRunnerTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/translation/ConfigGeneratorTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/translation/SamzaImpulseSystemTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/translation/TranslationContextTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/util/DoFnUtilsTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/util/FutureUtilsTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/util/InMemoryMetricsReporter.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/util/PipelineJsonRendererTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/util/PortableConfigUtilsTest.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/util/TestHashIdGenerator.java
delete mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/util/WindowUtilsTest.java
delete mode 100644 runners/samza/src/test/resources/ExpectedDag.json
delete mode 100644 runners/samza/src/test/resources/log4j-test.properties
create mode 100644 runners/spark/4/README.md
create mode 100644 runners/spark/4/build.gradle
rename runners/{samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaTranslatorRegistrar.java => spark/4/job-server/build.gradle} (62%)
copy runners/spark/{3 => 4}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/io/BoundedDatasetFactory.java (95%)
copy runners/spark/{3 => 4}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombinePerKeyTranslatorBatch.java (97%)
copy runners/spark/{3 => 4}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyHelpers.java (92%)
copy runners/spark/{3 => 4}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyTranslatorBatch.java (85%)
create mode 100644 runners/spark/4/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderFactory.java
copy runners/spark/{3 => 4}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpers.java (93%)
copy runners/spark/{3 => 4}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/ScalaInterop.java (94%)
copy runners/spark/{3 => 4}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpersTest.java (97%)
copy runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingPipelineOptions.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingPipelineResult.java (97%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunner.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunnerRegistrar.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/examples/WordCount.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/io/BoundedDatasetFactory.java (99%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/io/package-info.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/BeamMetricSet.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/MetricsAccumulator.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetric.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetricSource.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/WithMetricsSupport.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/package-info.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/CodahaleCsvSink.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/CodahaleGraphiteSink.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/package-info.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/package-info.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/EvaluationContext.java (93%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/SparkSessionFactory.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/SparkTransformOverrides.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/TransformTranslator.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/Aggregators.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombineGloballyTranslatorBatch.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombineGroupedValuesTranslatorBatch.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombinePerKeyTranslatorBatch.java (99%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DoFnPartitionIteratorFactory.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DoFnRunnerFactory.java (99%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DoFnRunnerWithMetrics.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/FlattenTranslatorBatch.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyHelpers.java (95%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyTranslatorBatch.java (99%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ImpulseTranslatorBatch.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ParDoTranslatorBatch.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/PipelineTranslatorBatch.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ReadSourceTranslatorBatch.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ReshuffleTranslatorBatch.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/WindowAssignTranslatorBatch.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/CachedSideInputReader.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/GroupAlsoByWindowViaOutputBufferFn.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/NoOpStepContext.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/SideInputValues.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/SparkSideInputReader.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/package-info.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/package-info.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/CoderHelpers.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderFactory.java (83%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpers.java (96%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderProvider.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/package-info.java (100%)
copy runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/package-info.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/ScalaInterop.java (100%)
rename runners/spark/{3 => }/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/package-info.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/SparkSessionRule.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunnerRegistrarTest.java (91%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/StructuredStreamingPipelineStateTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/InMemoryMetrics.java (73%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/InMemoryMetricsSinkRule.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/SparkMetricsSinkTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetricTest.java (100%)
create mode 100644 runners/spark/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/EvaluationContextTest.java
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/AggregatorsTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombineGloballyTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombineGroupedValuesTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombinePerKeyTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ComplexSourceTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/FlattenTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ParDoTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/SimpleSourceTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/WindowAssignTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/SideInputValuesTest.java (100%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpersTest.java (99%)
rename runners/spark/{3 => }/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/SimpleSourceTest.java (100%)
delete mode 100755 scripts/ci/release/comment_pr_trigger_phrases.sh
delete mode 100644 scripts/ci/release/test/resources/jenkins_trigger_phrases.txt
delete mode 100644 scripts/ci/release/test/resources/mass_comment.txt
create mode 100644 sdks/go/examples/ordered_list_state/ordered_list_state.go
create mode 100644 sdks/go/pkg/beam/artifact/options.go
create mode 100644 sdks/go/pkg/beam/artifact/options_test.go
delete mode 100644 sdks/go/pkg/beam/runners/samza/samza.go
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/lineage/LineageBase.java
rename runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingPipelineOptions.java => sdks/java/core/src/main/java/org/apache/beam/sdk/lineage/LineageOptions.java (51%)
copy {runners/samza/src/main/java/org/apache/beam/runners/samza/container => sdks/java/core/src/main/java/org/apache/beam/sdk/lineage}/package-info.java (61%)
copy sdks/java/core/src/main/java/org/apache/beam/sdk/{options/DefaultPipelineOptionsRegistrar.java => metrics/BoundedTrieMetricsLineage.java} (51%)
rename runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/SamzaDoFnInvokerRegistrar.java => sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/StringSetMetricsLineage.java (51%)
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/BatchElements.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/values/OpenTelemetryContextPropagator.java
rename runners/samza/src/main/java/org/apache/beam/runners/samza/transforms/UpdatingCombineFn.java => sdks/java/core/src/main/java/org/apache/beam/sdk/values/ValueKind.java (53%)
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/values/ValueKindUtil.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/lineage/LineagePluginTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/lineage/TestLineage.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/BatchElementsTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/MetadataPropagationTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ValueKindTest.java
delete mode 100644 sdks/java/extensions/avro/vendored-test/build.gradle
copy sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/{ClickHouseIOTest.java => ClickHouseIOIT.java} (93%)
create mode 100644 sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/FileSystemOffsetRetainer.java
create mode 100644 sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/OffsetRetainer.java
create mode 100644 sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/FileSystemOffsetRetainerTest.java
rename runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OpEmitter.java => sdks/java/io/delta/build.gradle (54%)
create mode 100644 sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta/DeltaIO.java
rename {runners/samza/src/main/java/org/apache/beam/runners/samza/metrics => sdks/java/io/delta/src/main/java/org/apache/beam/sdk/io/delta}/package-info.java (85%)
create mode 100644 sdks/java/io/delta/src/test/java/org/apache/beam/sdk/io/delta/DeltaIOTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AppendClientCache.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/ConvertMessagesDoFn.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/MergeSchemaCombineFn.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/PatchTableSchemaDoFn.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/SchemaUpdateHoldingFn.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/UpgradeTableSchema.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableRowEqualityTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiDataTriggeredSchemaUpdateIT.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowMatchers.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/UpgradeTableSchemaTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessagesTest.java
create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFiles.java
create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFilesSchemaTransformProvider.java
create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AssignDestinationsAndPartitions.java
create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/BeamRowWrapper.java
create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SortOrderUtils.java
create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/WritePartitionedRowsToFiles.java
create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/WriteToPartitions.java
create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/AddFilesIT.java
create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/AddFilesTest.java
create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/BeamRowWrapperTest.java
create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/SortOrderUtilsTest.java
delete mode 100644 sdks/python/.isort.cfg
create mode 100644 sdks/python/apache_beam/examples/inference/table_row_batch_example.py
create mode 100644 sdks/python/apache_beam/examples/inference/table_row_inference.py
create mode 100644 sdks/python/apache_beam/examples/inference/table_row_inference_test.py
create mode 100644 sdks/python/apache_beam/examples/inference/table_row_inference_utils.py
create mode 100644 sdks/python/apache_beam/io/gcp/bigquery_change_history.py
create mode 100644 sdks/python/apache_beam/io/gcp/bigquery_change_history_it_test.py
create mode 100644 sdks/python/apache_beam/io/gcp/bigquery_change_history_test.py
create mode 100644 sdks/python/apache_beam/ml/inference/agent_development_kit.py
create mode 100644 sdks/python/apache_beam/ml/inference/agent_development_kit_test.py
create mode 100644 sdks/python/apache_beam/ml/inference/anthropic_inference.py
create mode 100644 sdks/python/apache_beam/ml/inference/anthropic_inference_it_test.py
create mode 100644 sdks/python/apache_beam/ml/inference/anthropic_inference_test.py
copy sdks/python/apache_beam/{version.py => ml/inference/anthropic_tests_requirements.txt} (90%)
copy sdks/python/apache_beam/ml/inference/{vllm_tests_requirements.txt => table_row_inference_requirements.txt} (90%)
create mode 100644 sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/install.json
create mode 100644 sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel/style/index.js
delete mode 100644 sdks/python/apache_beam/runners/portability/samza_runner_test.py
create mode 100644 sdks/python/apache_beam/testing/benchmarks/inference/table_row_inference_benchmark.py
create mode 100644 sdks/python/apache_beam/testing/benchmarks/sort_and_batch_benchmark.py
create mode 100644 sdks/python/apache_beam/typehints/row_type_test.py
copy sdks/python/apache_beam/{examples/ml-orchestration/kfp/components/preprocessing/requirements.txt => utils/byte_limited_queue.pxd} (64%)
create mode 100644 sdks/python/apache_beam/utils/byte_limited_queue.py
create mode 100644 sdks/python/apache_beam/utils/byte_limited_queue_test.py
create mode 100644 sdks/python/apache_beam/yaml/extended_tests/databases/bigquery_dynamic_destinations.yaml
create mode 100644 sdks/python/apache_beam/yaml/tests/iceberg_add_files.yaml
create mode 100644 sdks/python/apache_beam/yaml/tests/iceberg_add_files_batch.yaml
create mode 100644 sdks/python/apache_beam/yaml/tests/reshuffle.yaml
rename runners/samza/src/main/java/org/apache/beam/runners/samza/container/package-info.java => sdks/python/container/distroless/py314/build.gradle (70%)
copy runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/package-info.java => sdks/python/container/ml/py314/build.gradle (70%)
copy sdks/python/container/{py313 => py314}/base_image_requirements.txt (76%)
rename runners/samza/src/main/java/org/apache/beam/runners/samza/adapter/package-info.java => sdks/python/container/py314/build.gradle (71%)
delete mode 100644 sdks/python/mypy.ini
delete mode 100644 sdks/python/pyrefly.toml
rename sdks/python/scripts/{run_pylint.sh => run_lint.sh} (91%)
copy runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/package-info.java => sdks/python/test-suites/dataflow/py314/build.gradle (68%)
copy runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/package-info.java => sdks/python/test-suites/direct/py314/build.gradle (69%)
rename runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/package-info.java => sdks/python/test-suites/portable/py314/build.gradle (66%)
copy sdks/python/test-suites/tox/{pycommon => py314}/build.gradle (66%)
create mode 100644 website/www/site/content/en/blog/beam-2.72.0.md
create mode 100644 website/www/site/content/en/blog/beam-2.73.0.md
create mode 100644 website/www/site/content/en/documentation/io/built-in/iceberg.md
create mode 100644 website/www/site/content/en/documentation/patterns/rate-limiting.md
create mode 100644 website/www/site/content/en/documentation/transforms/java/aggregation/batchelements.md
create mode 100644 website/www/site/content/en/performance/tablerowinference/_index.md
create mode 100644 website/www/site/content/en/performance/tablerowinferencestreaming/_index.md
copy website/www/site/layouts/shortcodes/{button-colab.html => section.html} (64%)
copy website/www/site/layouts/shortcodes/{button-colab.html => tab.html} (57%)
create mode 100644 website/www/site/static/images/colab/icon32.png
(beam) 01/01: Allow regeneration.
Posted by tv...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 8cd9163fc7a9173f0bc339a479d3db2b6cb7a052
Author: Valentyn Tymofieiev <va...@google.com>
AuthorDate: Fri May 22 15:17:16 2026 -0700
Allow regeneration.
---
.github/workflows/update_python_dependencies.yml | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/.github/workflows/update_python_dependencies.yml b/.github/workflows/update_python_dependencies.yml
index 753fb6037fc..6fc7de67cd9 100644
--- a/.github/workflows/update_python_dependencies.yml
+++ b/.github/workflows/update_python_dependencies.yml
@@ -83,10 +83,6 @@ jobs:
# then get the num days % 42 (our release cadence is 42 days).
# This will ensure it only runs the week after a release branch has been cut.
days_diff=$(( ($(date +%s) - $(date --date="240306" +%s) )/(60*60*24)%42 ))
- if [[ $GH_EVENT != 'workflow_dispatch' && $days_diff -gt 6 ]]; then
- echo "Exiting early. We only update dependencies the week after we cut the release"
- exit 0
- fi
branchName=weekly_update_python_dependencies_$(date +%s)
git checkout -b $branchName
git add -A
@@ -95,7 +91,7 @@ jobs:
echo "Changes are ready to commit"
git commit -m "Update Python Dependencies" --quiet
git push origin $branchName --quiet
- GITHUB_PR_URL=$(gh pr create --title "Update Python Dependencies" --label "python" --body "This PR was created by automation - by default this does not execute tests due to a limitation in GitHub Actions. To review this PR, please (1) review the contents to make sure all dependency upgrades are formatted correctly, (2) close and reopen the PR (3) Approve/merge if tests pass successfully.")
+ GITHUB_PR_URL=$(gh pr create --title "Update Python Dependencies - manual trigger" --label "python" --body "This PR was created by automation - by default this does not execute tests due to a limitation in GitHub Actions. To review this PR, please (1) review the contents to make sure all dependency upgrades are formatted correctly, (2) close and reopen the PR (3) Approve/merge if tests pass successfully.")
echo "Link of the new PR $GITHUB_PR_URL"
else
echo "No changes on the files"