You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2024/09/13 22:24:15 UTC
(beam) branch dependabot/go_modules/sdks/github.com/fsouza/fake-gcs-server-1.49.3 updated (a14f842fd6e -> a9ea4fe12a7)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/go_modules/sdks/github.com/fsouza/fake-gcs-server-1.49.3
in repository https://gitbox.apache.org/repos/asf/beam.git
discard a14f842fd6e Bump github.com/fsouza/fake-gcs-server from 1.49.2 to 1.49.3 in /sdks
add a2f5ee26941 Fix configuration of ReadFromKafkaViaSDF which was always enabling redistribute and allowing duplicates instead of basing it on configuration. (#32134)
add 8171e8c29a0 copy editing the Gemma notebook (#32188)
add d716f3152a3 Increase snapshot publish timeout (#32197)
add c197e4ffc1a [#32121] Support timers in interval windows. (#32180)
add 028e0eef45a Protect release branch (#32204)
add d7d9f5145a3 Fix tests after GHA runner Java11 migration (#32190)
add 2d140762eaa Add callback to with_exception_handling (#32136)
add 92087f2daa7 [prism] Catch panics in primary execution goroutines. (#32210)
add cbe2b9e2dde [#31926] [java] call provision service when creating external workers. (#32198)
add 11befd32662 [#31927,#31928][prism] Support StringSet and Gauge metrics. (#32184)
add ced67ec1100 [Java11 Migration] Migrate Go testing to use Java11 container image for xlang (#32212)
add 65550a71f96 [yaml] Doc improvements (#32117)
add ff93c48a4c5 Update Gemma 2 Notebook (#32200)
add f1e214712c8 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#32214)
add c9ad32ea8f4 [Managed Iceberg] Support writing to partitioned tables (#32102)
add 83a3db5ce7e More Java11 test fix (#32209)
add 307002e0146 Update website for 2.58.1 release (#32205)
add 541e5c85031 Make BQ file load limit controls public (#32101)
add 391dbf6b243 SolaceIO.read: Encode url parameters in REST requests. (#32133)
add e6c42d2dd82 update runner image (#32220)
add e4f2322453d Remove executable non-python blocks from Gemma 2 notebook (#32219)
add 8c0fbf75c3a Replacing Jackson Factory with Gson Factory and further code updation (#32158)
add 88713ace637 [prism] rename watermarkRefreshs concept to changedStages. (#32233)
add aa13a692cd8 Add hard delete function to multi_process_shared (#32238)
add 872a97f15d5 [#32080] Remove restriction on requests package (#32236)
add bed2e562ece Update ClickHouseIO to use the latest version of the ClickHouse JDBC driver (0.6.4) (#32229)
add 6b4a7a5d73e feat: optimize Spanner changestream metadata table (#32213)
add e2bf5d6dbe3 Add basic testing for yaml join docs.
add 7f1c7f4d8df make linter happy
add 38dfbd4c35e Merge pull request #32141 Add basic testing for yaml join docs.
add 106ba39a3fd Update beam-master to 20240819 (#32240)
add 9888900b5db Include Iceberg Hive runtime dependencies in IO expansion service container (#32232)
add 68ddd9dc7b9 add a pattern for using a shared object as a cache
add afb6a09094b apply suggestions from code review
add 049b478ddd9 fix whitespace error
add d34c927e57e Apply suggestions from code review
add 067d6b8edc6 Merge pull request #32187 from jaehyeon-kim/feature/add-shared-class-examples
add 26cd5df5995 Adds an ORDERED_LIST_STATE capability to the Java SDK. (#32067)
add ff64566fbdd Bump google-ads API to v17 (#32244)
add 714f08b34ae Add to ClickHouseIO dedicated useragnet (#32253)
add 6582e7ae538 Fix it-mongo compile fail on Java11 (#32243)
add bd65ee9c5bf Portable runner fixes (#32247)
add 254519b857a Add Lineage metrics to KafkaIO (#32170)
add 89b1a7f2028 [yaml] Fix PubSub error message
add 4365f73cbe3 Merge pull request #32093 [yaml] Fix PubSub error message
add 63055a8032f Add BatchElements overview doc to Beam Website (#32239)
add 2da24d0644f [#32245][Go SDK] Copy bytes sent over the State API Writer. (#32246)
add 71e3eedcd65 Update nltk version to 3.9.1 (#32256)
add 05b1781c6ea [#32221] [prism] Terminate streams for each timerfamily+transform pair. (#32223)
add ec3cec201bf Attach file extension to Iceberg writes (#32254)
add 65427efe9d0 Bump cloud.google.com/go/spanner from 1.66.0 to 1.67.0 in /sdks (#32234)
add 228554b7319 Bump google.golang.org/api from 0.191.0 to 0.192.0 in /sdks (#32175)
add 8601bbaed8d Bump github.com/tetratelabs/wazero from 1.7.3 to 1.8.0 in /sdks (#32193)
add 3fb4fd0f6ea Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#32266)
add 917e99670ed Add ErrorHandler pattern to Python.
add c4be92fd769 Add with_error_handler to ParDo (Map, FlatMap, etc.)
add 34e28f396ea Add collecting error handler.
add b69f4d8eb45 Add test stanza and other lint fixes.
add daf28cdf979 Fix typo.
add 5141f14503d Fix typo.
add 049e4b3b6b9 Merge branch 'master' into error-handler
add 36e5eff40af Add test of with_exception_handling side effects.
add b3a874f4766 Merge pull request #31856 Add ErrorHandler DLQ API to Python.
add 3bf24217c79 [Managed Iceberg] Support BigQuery Metastore catalog (#32242)
add a0e4541f0f8 Allow setting BigQuery endpoint (#32153)
add b873be8bdb9 Add timeout to runinference (#32237)
add c3432b73f4e Bump github.com/testcontainers/testcontainers-go in /sdks (#32267)
add 37ad3196d81 Fixes a breakage related to Kafka upgrade (#32262)
add c5bf5fc7395 Revert "Portable runner fixes (#32247)" (#32271)
add 65f556d861e Update requests package to 2.32.0 (#32270)
add 512b52a2b23 Better error when Python xlang download jar not available (#32269)
add ed4c03e3779 Do not create new Executor everytime createRunner (#32272)
add a62ff34b7cf Better error message when trying to get Beam Schema from proto schema with Struct (#32260)
add 24699908ebc Moving to 2.60.0-SNAPSHOT on master branch.
add 6318bc3461a add set state in spark runner (#32226)
add 995724d651a [prism] Add an idle shutdown timout to prism binary. (#32276)
add 8488a0d056d Move RunInference GC to finalize_bundle (#32281)
add ea882ab55b6 Check experiment for hotkey logging (#32285)
add ebf80f326b2 download BQMS catalog jar at build time (#32282)
add e62daf70077 [CHANGES.md] Add 2.60.0 section. (#32287)
add 22fbdd5a849 Disable sickbay scheduled runs (#32299)
add b2ed1c56402 Loose BigQuery GCP project ID regex restrictions (#32178)
add 1a8ef79eb35 Remove some setup steps we don't need anymore (#32300)
add 1e80815d12d Parallelize building wheels per language version (#32297)
add c77460e32b9 Rename delimiter to sep to pass to pandas.
add e24a0c473fd Merge pull request #32301 Rename delimiter to sep to pass to pandas.
add 46cdc4b220c Fix split with delimiter (#32298)
add b8bbf593abf Implement Java TestPrismRunner and PrismRunner (#32294)
add 9c0a9503ebd Make autosharding test more robust (#32293)
add b23d3ed68b3 patch release docs (#32318)
add dd662f639b2 fixed invalid links on programming-guide.md (#32317)
add 3dec995d109 Add beam summit banner (#32320)
add 24255ac84b7 Add schema to SpannerIO read (#32008)
add 0ae3b13ac61 Move beam summit banner to top of swiper slide
add 43963b6f07f Merge pull request #32323 from apache/users/damccorm/summitBanner
add d1157d8a9c0 fix failing ProtoSchemaTranslator on proto3 optional fields (#32216)
add 857eccedc55 Refactor PubsubIO Lineage metrics to work with all runners (#32319)
add 142e39250db Preserve numeric string literals when reading from json.
add 679e5cc6ff8 Add a test.
add bc80e9fdbb2 Merge pull request #32303 Preserve numeric string literals when reading from json.
add 9e3aeca6123 Add doc string and signature to generated Python wrappers (#32337)
add 8cc80ff44c5 Add quality warnings to pulsar (#32346)
add 815a049b1bd Document dynamic destination key should be unique over the pipeline (#32338)
add a85b0a636dc Filter out old actions runs from dashboard (#32347)
add 7d5c97320b9 Add predicate to control which columns are propagated when propagating successful rows (#32312)
add a6cef9210b9 Fix remote execution test flakiness in tearDown (#32328)
add 399112b2fab Fix BigtableIO.write() client sharing (#32340)
add 923fde0f070 Fix pr building of wheels (#32353)
add 55678b2edcf Pause delete images in public AR (#32354)
add 7d6f6fb55bc Try deflaking test timing (#32351)
add d3f38bec406 Update Python Dependencies (#32310)
add bcee5d081d0 Remove expensive shuffle of read data in KafkaIO when using sdf and commit offsets (#31682)
add df83fbda179 Relocate guava in shaded hive-exec (#32363)
add 2ac5594ce8f add ExternalTransformProvider example (#30666)
add a8954691239 Updates Python SDK harness HEAD container for Dataflow. (#32369)
add 28f2d47662a docs: modernize py dependencies docs and example (#32345)
add 322c47da41b Fix Dataflow ARM Java Example PostCommit bad gradle cache (#32356)
add 66bc071fd5e Avoid parsing all options just to grab an option that is already parsed.
add 8fe3386c847 The GCS options are in GoogleCloudOptions, not generic PipelineOptions
add b0caa7976c3 Fix formatting issues
add aac398dab5c Fix formatting issues (again)
add fd8d14660af Merge pull request #32362 from iht/fix_gcsio_options
add 52ab49e7255 Fix BQ storage stream split (#32376)
add cfe8feee7c5 Improve BatchElements documentation (#32082)
add 511f294db15 Revert "docs: modernize py dependencies docs and example" (#32382)
add e18596c7fc4 Temporarily remove BQMS catalog until it is opend-sourced (#32386)
add 64b4aa6fdde Memoize toString() for MetricKey and MetricName (#32379)
add 9771a425655 Add WindowedvalueParam option to DoFns.
add b875fd490d9 Merge pull request #32305 Add WindowedValueParam option to DoFns.
add 6baba928348 [YAML] Better errors for unconsumed error outputs. (#32341)
add f8bda18c737 Use proper coders in interactive cache.
add f06df5dabf0 Merge pull request #32330 Use proper coders in interactive cache.
add 2b02fd3cc94 Allow ib.collect to work with non-interactive runners. (#32383)
add 34a7c4f4198 Upgrade artifact actions to v4 (#32391)
add 696978018ed Bump actions/download-artifact from 4.1.7 to 4.1.8 (#32396)
add 3b4ecd7a987 Bump commons-cli:commons-cli from 1.8.0 to 1.9.0 (#32350)
add 4dddde4e6de Colab Notebook for Unit Tests in Beam (#32336)
add 6901d7c8623 Undo part of artifact action upgrade to fix workflow (#32401)
add 1ee2f6bff4a Bump jinja2 version to resolve vulnerability (#32403)
add 50a6cd2b580 Add tests of using ib.collect(...) without InteractiveRunner.
add d5dc20000d5 Allow ib.collect(...) to take multiple PCollections.
add a7852d9ddb1 Merge pull request #32392 Allow ib.collect(...) to take multiple PCollections.
add d9a3d4457a0 Add keyrings.google-artifactregistry-auth (#32404)
add 2a7755b0f2c Use Knuth–Morris–Pratt algorithm for delimiter search in TextIO (#32398)
add 7a88e6fa0bb [Dataflow Streaming] Move `throwExceptionOnLargeOutput` out of OperationalLimits (#32407)
add 173dd486842 [yaml] Adding Spanner IO Providers for Beam YAML (#31987)
add 4aeb246596d [yaml] Move js2py package to yaml deps (#32377)
add 2573106f367 Bump webpack (#32374)
add 284004484c1 Add jobLabelsMap parameter to BigQueryOptions (#31698)
add 1d0e09a6d57 Support Zstd codec in SerializableAvroCodecFactory (#32352)
add f362045afb7 Bump micromatch from 4.0.2 to 4.0.8 in /website/www
add 2a0ce8f9103 Merge pull request #32375 from apache/dependabot/npm_and_yarn/website/www/micromatch-4.0.8
add 5fbfdba926c Move PubsubIO source Lineage report to MapElements (#32381)
add c657507c082 Upgrade gcp-bom to 26.45.0 (#32413)
add ebcb2dbd160 Dont run flaky test on windows (#32419)
add a806bfc0526 [Python] Add the "no_wait_until_finish" option without waiting fo the job completion
add e8c6a8c3aec Merge pull request #32409 from liferoad/wait_until_finish
add e76b51feb3a Fix RowCoderGenerator to use the encodingPositions when encoding and decoding the bit set representing null fields. (#32389)
add 21abfb23f36 Disable permared go xlang tests. (#32423)
add 9c31e45d595 2.59.0 Beam site updates. (#32308)
add ef1bb52d3d3 Fixes link typo in 2.59, and fixes archive links. (#32437)
add 17718a97eaf Temporarily fix pip check failure on xgboost and grpcio (#32432)
add ed84acb2967 Bump spanner max bound back to <4 (#32411)
add 4ee26065d9b Accept runner and options in ib.collect. (#32434)
add 02af7d4a218 Unit Testing in Beam Blog Post (#32412)
add bcf404238ab downgrade grpc (#32449)
add 45036736e0a Remove beam summit banner
add 98410caac19 Merge pull request #32331 from apache/users/damccorm/beamsummit
add 9a524cc004e Add Lineage metrics to Python PubsubIO, BigtableIO, FileIO (#32430)
add 83ae551cea1 Update Playground and other location usage of Go to use latest released version by default. (#32444)
add ed8dc5df618 fix: use threadsafe data structure thats shared between threads (#32452)
add 25196614070 fix: ensure that non ApiExceptions in handleMutationException are not swallowed (#32455)
add a6f11c89667 Bump Python SDK container for Dataflow
add 1c307088a86 Merge pull request #32453: Bump Python SDK container for Dataflow
add fb5424e5888 Bump cloud.google.com/go/pubsub from 1.41.0 to 1.43.0 in /sdks (#32415)
add a9ea4fe12a7 Bump github.com/fsouza/fake-gcs-server from 1.49.2 to 1.49.3 in /sdks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a14f842fd6e)
\
N -- N -- N refs/heads/dependabot/go_modules/sdks/github.com/fsouza/fake-gcs-server-1.49.3 (a9ea4fe12a7)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 2 +
.../arc/environments/beam.env | 6 +-
.../IO_Iceberg_Integration_Tests.json | 2 +-
...am_PostCommit_Java_Examples_Dataflow_Java.json} | 0
...pendency.json => beam_PreCommit_Python_ML.json} | 0
.github/workflows/beam_PostCommit_Go_VR_Samza.yml | 5 +
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 6 +-
.../beam_PostCommit_Java_Hadoop_Versions.yml | 14 +-
.../workflows/beam_PostCommit_Java_PVR_Samza.yml | 8 +
.../beam_PostCommit_Java_ValidatesRunner_Samza.yml | 8 +-
...eam_PostCommit_Python_ValidatesRunner_Samza.yml | 3 +-
.../workflows/beam_PostCommit_Sickbay_Python.yml | 4 +-
.github/workflows/beam_PostCommit_XVR_Samza.yml | 2 +
.../beam_PreCommit_Java_HCatalog_IO_Direct.yml | 15 +
.../workflows/beam_PreCommit_Java_IOs_Direct.yml | 15 +
.../beam_PreCommit_Java_Spark3_Versions.yml | 10 +-
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 2 +-
.github/workflows/build_wheels.yml | 55 +-
.github/workflows/python_tests.yml | 10 +-
.test-infra/dockerized-jenkins/plugins.txt | 2 +-
.../github/github_runs_prefetcher/code/main.py | 3 +
.../tools/stale_dataflow_prebuilt_image_cleaner.sh | 2 +-
CHANGES.md | 71 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 33 +-
contributor-docs/release-guide.md | 143 +-
examples/multi-language/README.md | 3 +
.../multi-language/python/wordcount_external.py | 118 ++
.../schematransforms/ExtractWordsProvider.java | 76 +
.../schematransforms/JavaCountProvider.java | 78 +
.../schematransforms/WriteWordsProvider.java | 77 +
.../gemma_2_sentiment_and_summarization.ipynb | 48 +-
examples/notebooks/blog/unittests_in_beam.ipynb | 370 ++++
gradle.properties | 4 +-
.../it/cassandra/CassandraResourceManager.java | 2 +-
.../apache/beam/it/common/PipelineLauncher.java | 2 +-
.../org/apache/beam/it/common/TestProperties.java | 2 +-
.../beam/it/common/utils/ExceptionUtils.java | 2 +-
.../beam/it/common/utils/ResourceManagerUtils.java | 2 +-
.../apache/beam/it/conditions/ConditionCheck.java | 2 +-
.../ElasticsearchResourceManager.java | 2 +-
.../java/org/apache/beam/it/gcp/LoadTestBase.java | 2 +-
.../gcp/bigquery/conditions/BigQueryRowsCheck.java | 5 +-
.../it/gcp/bigtable/BigtableResourceManager.java | 2 +-
.../it/gcp/dataflow/AbstractPipelineLauncher.java | 2 +-
.../beam/it/gcp/monitoring/MonitoringClient.java | 2 +-
.../gcp/pubsub/conditions/PubsubMessagesCheck.java | 5 +-
.../it/gcp/spanner/SpannerResourceManager.java | 2 +-
.../beam/it/gcp/bigquery/BigQueryStreamingLT.java | 3 +-
.../apache/beam/it/kafka/KafkaResourceManager.java | 2 +-
it/mongodb/build.gradle | 4 +
.../beam/it/mongodb/MongoDBResourceManager.java | 2 +-
.../mongodb/conditions/MongoDBDocumentsCheck.java | 5 +-
.../apache/beam/it/neo4j/Neo4jResourceManager.java | 2 +-
.../beam/it/neo4j/conditions/Neo4jQueryCheck.java | 5 +-
.../beam/it/splunk/SplunkResourceManager.java | 2 +-
.../it/splunk/conditions/SplunkEventsCheck.java | 8 +-
.../TestContainerResourceManager.java | 2 +-
.../beam/it/truthmatchers/PipelineAsserts.java | 2 +-
.../beam/it/truthmatchers/RecordsSubject.java | 2 +-
.../beam/model/pipeline/v1/beam_runner_api.proto | 4 +
playground/README.md | 2 +-
playground/backend/containers/go/build.gradle | 2 +-
playground/backend/containers/java/Dockerfile | 2 +-
playground/backend/containers/python/Dockerfile | 2 +-
playground/backend/containers/python/build.gradle | 2 +-
playground/backend/containers/router/Dockerfile | 2 +-
playground/backend/containers/router/build.gradle | 2 +-
playground/backend/containers/scio/Dockerfile | 2 +-
playground/backend/playground_functions/Dockerfile | 2 +-
.../cloudbuild/playground_ci_examples.sh | 4 +-
release/build.gradle.kts | 4 +-
release/go-licenses/Dockerfile | 2 +-
.../wrappers/streaming/SplittableDoFnOperator.java | 9 +-
runners/google-cloud-dataflow-java/build.gradle | 4 +-
.../examples/build.gradle | 8 +-
.../dataflow/DataflowPipelineTranslator.java | 2 +-
.../runners/dataflow/util/PackageUtilTest.java | 4 +-
.../worker/DataflowWorkProgressUpdater.java | 3 +-
.../runners/dataflow/worker/OperationalLimits.java | 14 +-
.../worker/SplittableProcessFnFactory.java | 15 +-
.../dataflow/worker/StreamingDataflowWorker.java | 5 -
.../worker/StreamingModeExecutionContext.java | 7 +-
.../processing/ComputationWorkExecutorFactory.java | 10 +-
.../work/processing/StreamingWorkScheduler.java | 5 +-
.../worker/StreamingDataflowWorkerTest.java | 15 +-
.../worker/StreamingModeExecutionContextTest.java | 3 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 6 +-
.../graph/LengthPrefixUnknownCodersTest.java | 10 +-
.../dataflow/worker/testing/GenericJsonAssert.java | 8 +-
.../worker/testing/GenericJsonMatcher.java | 8 +-
.../fnexecution/control/RemoteExecutionTest.java | 10 +-
runners/prism/java/build.gradle | 1 +
.../apache/beam/runners/prism/PrismExecutor.java | 5 +-
.../apache/beam/runners/prism/PrismLocator.java | 12 +-
.../beam/runners/prism/PrismPipelineOptions.java | 17 +
.../beam/runners/prism/PrismPipelineResult.java | 33 +-
.../org/apache/beam/runners/prism/PrismRunner.java | 72 +-
.../beam/runners/prism/PrismRunnerRegistrar.java | 25 +-
.../runners/prism/TestPrismPipelineOptions.java | 8 +-
.../apache/beam/runners/prism/TestPrismRunner.java | 84 +
.../beam/runners/prism/PrismExecutorTest.java | 8 +-
.../runners/prism/PrismPipelineResultTest.java | 130 --
.../apache/beam/runners/prism/PrismRunnerTest.java | 117 +-
.../SplittableParDoProcessKeyedElementsOp.java | 12 +-
.../spark/stateful/SparkStateInternals.java | 78 +-
.../spark/stateful/SparkStateInternalsTest.java | 26 +-
scripts/tools/bomupgrader.py | 2 +-
sdks/go.mod | 110 +-
sdks/go.sum | 226 +-
sdks/go/cmd/prism/prism.go | 25 +-
sdks/go/examples/kafka/taxi.go | 6 +-
sdks/go/examples/xlang/bigquery/wordcount.go | 6 +-
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/go/pkg/beam/core/runtime/harness/statemgr.go | 9 +-
.../beam/model/fnexecution_v1/beam_fn_api.pb.go | 7 +-
.../model/fnexecution_v1/beam_fn_api_grpc.pb.go | 2 +-
.../model/fnexecution_v1/beam_provision_api.pb.go | 2 +-
.../fnexecution_v1/beam_provision_api_grpc.pb.go | 2 +-
.../model/jobmanagement_v1/beam_artifact_api.pb.go | 2 +-
.../jobmanagement_v1/beam_artifact_api_grpc.pb.go | 2 +-
.../jobmanagement_v1/beam_expansion_api.pb.go | 2 +-
.../jobmanagement_v1/beam_expansion_api_grpc.pb.go | 2 +-
.../beam/model/jobmanagement_v1/beam_job_api.pb.go | 2 +-
.../model/jobmanagement_v1/beam_job_api_grpc.pb.go | 2 +-
.../beam/model/pipeline_v1/beam_runner_api.pb.go | 2177 ++++++++++----------
.../model/pipeline_v1/beam_runner_api_grpc.pb.go | 2 +-
sdks/go/pkg/beam/model/pipeline_v1/endpoints.pb.go | 2 +-
.../model/pipeline_v1/external_transforms.pb.go | 206 +-
sdks/go/pkg/beam/model/pipeline_v1/metrics.pb.go | 698 ++++---
sdks/go/pkg/beam/model/pipeline_v1/schema.pb.go | 2 +-
.../model/pipeline_v1/standard_window_fns.pb.go | 2 +-
sdks/go/pkg/beam/runners/prism/internal/coders.go | 12 +-
.../pkg/beam/runners/prism/internal/coders_test.go | 16 +-
.../prism/internal/engine/elementmanager.go | 136 +-
.../runners/prism/internal/engine/teststream.go | 12 +-
.../beam/runners/prism/internal/engine/timers.go | 68 +-
sdks/go/pkg/beam/runners/prism/internal/execute.go | 39 +-
.../beam/runners/prism/internal/handlerunner.go | 2 +-
.../prism/internal/jobservices/management.go | 22 +-
.../runners/prism/internal/jobservices/metrics.go | 123 +-
.../prism/internal/jobservices/metrics_test.go | 38 +
.../runners/prism/internal/jobservices/server.go | 53 +-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 56 +-
sdks/go/pkg/beam/runners/prism/internal/web/web.go | 8 +-
.../beam/runners/prism/internal/worker/bundle.go | 5 +-
sdks/go/pkg/beam/runners/prism/prism.go | 12 +
sdks/go/test/build.gradle | 10 +-
.../resources/beam/checkstyle/suppressions.xml | 4 +-
.../container/license_scripts/dep_urls_java.yaml | 2 +-
.../java/org/apache/beam/sdk/coders/RowCoder.java | 8 +-
.../apache/beam/sdk/coders/RowCoderGenerator.java | 131 +-
.../java/org/apache/beam/sdk/io/TextSource.java | 220 +-
.../java/org/apache/beam/sdk/metrics/Lineage.java | 18 +-
.../org/apache/beam/sdk/metrics/MetricKey.java | 4 +-
.../org/apache/beam/sdk/metrics/MetricName.java | 2 +
.../org/apache/beam/sdk/schemas/SchemaCoder.java | 7 +-
.../beam/sdk/schemas/transforms/CoGroup.java | 7 +-
.../transforms/providers/ErrorHandling.java | 3 +-
.../java/org/apache/beam/sdk/transforms/Watch.java | 11 +-
.../beam/sdk/util/construction/Environments.java | 1 +
.../org/apache/beam/sdk/coders/RowCoderTest.java | 134 +-
.../org/apache/beam/sdk/io/TextIOReadTest.java | 75 +-
.../org/apache/beam/sdk/io/TextSourceTest.java | 156 ++
.../sdk/util/construction/EnvironmentsTest.java | 3 +
.../avro/io/SerializableAvroCodecFactory.java | 12 +-
.../avro/io/SerializableAvroCodecFactoryTest.java | 48 +-
.../google-cloud-platform-core/build.gradle | 2 +-
.../beam/sdk/extensions/gcp/util/Transport.java | 4 +-
.../beam/sdk/extensions/gcp/util/GcsUtilTest.java | 10 +-
...LatencyRecordingHttpRequestInitializerTest.java | 4 +-
.../gcp/util/RetryHttpRequestInitializerTest.java | 4 +-
.../extensions/protobuf/ProtoSchemaTranslator.java | 12 +-
.../protobuf/ProtoSchemaTranslatorTest.java | 15 +
.../sdk/extensions/protobuf/TestProtoSchemas.java | 22 +
.../src/test/proto/proto3_schema_messages.proto | 14 +-
.../src/test/proto/proto3_schema_options.proto | 2 +
.../beam/fn/harness/ExternalWorkerService.java | 44 +-
sdks/java/io/clickhouse/build.gradle | 2 +-
.../beam/sdk/io/clickhouse/ClickHouseIO.java | 9 +-
.../beam/sdk/io/clickhouse/ClickHouseWriter.java | 6 +-
.../beam/sdk/io/clickhouse/ClickHouseIOTest.java | 29 +-
sdks/java/io/expansion-service/build.gradle | 7 +-
sdks/java/io/google-ads/build.gradle | 2 +-
.../apache/beam/sdk/io/googleads/GoogleAdsIO.java | 8 +-
.../{GoogleAdsV14.java => GoogleAdsV17.java} | 100 +-
.../sdk/io/googleads/DummyRateLimitPolicy.java | 11 +-
...GoogleAdsV14Test.java => GoogleAdsV17Test.java} | 90 +-
.../io/googleads/MockGoogleAdsClientFactory.java | 10 +-
sdks/java/io/google-cloud-platform/build.gradle | 3 +-
.../beam/sdk/io/gcp/bigquery/AppendClientInfo.java | 6 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 75 +-
.../sdk/io/gcp/bigquery/BigQueryIOTranslation.java | 11 +
.../beam/sdk/io/gcp/bigquery/BigQueryOptions.java | 18 +
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 92 +-
.../gcp/bigquery/BigQueryStorageStreamSource.java | 8 +-
.../sdk/io/gcp/bigquery/DynamicDestinations.java | 4 +
.../StorageApiDynamicDestinationsProto.java | 4 +-
.../beam/sdk/io/gcp/bigquery/StorageApiLoads.java | 7 +
.../StorageApiWriteRecordsInconsistent.java | 7 +
.../bigquery/StorageApiWriteUnshardedRecords.java | 24 +-
.../bigquery/StorageApiWritesShardedRecords.java | 17 +-
.../io/gcp/bigquery/TableRowToStorageApiProto.java | 30 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 111 +-
.../io/gcp/bigtable/BigtableServiceFactory.java | 9 +
.../io/gcp/healthcare/HttpHealthcareApiClient.java | 4 +-
.../sdk/io/gcp/pubsub/PreparePubsubWriteDoFn.java | 10 +
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 56 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSink.java | 12 -
.../sdk/io/gcp/pubsub/PubsubUnboundedSource.java | 14 -
.../beam/sdk/io/gcp/spanner/MutationUtils.java | 60 +
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 66 +-
.../sdk/io/gcp/spanner/SpannerQuerySourceDef.java | 56 +
.../SpannerReadSchemaTransformProvider.java | 235 +++
.../spanner/SpannerSchemaRetrievalException.java} | 14 +-
.../beam/sdk/io/gcp/spanner/SpannerSourceDef.java} | 25 +-
.../sdk/io/gcp/spanner/SpannerTableSourceDef.java | 63 +
.../SpannerWriteSchemaTransformProvider.java | 160 +-
.../beam/sdk/io/gcp/spanner/StructUtils.java | 67 +
.../dao/PartitionMetadataAdminDao.java | 83 +-
.../sdk/io/gcp/testing/FakeDatasetService.java | 5 +-
.../sdk/io/gcp/bigquery/BigQueryHelpersTest.java | 1 +
.../io/gcp/bigquery/BigQueryIOTranslationTest.java | 3 +
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 103 +
.../io/gcp/bigquery/BigQueryServicesImplTest.java | 33 +-
.../io/gcp/bigtable/BigtableSharedClientTest.java | 271 +++
.../beam/sdk/io/gcp/healthcare/FhirIOTestUtil.java | 5 +-
.../beam/sdk/io/gcp/spanner/SpannerIOReadTest.java | 26 +
.../beam/sdk/io/gcp/spanner/SpannerReadIT.java | 34 +
.../beam/sdk/io/gcp/spanner/StructUtilsTest.java | 66 +
.../dao/PartitionMetadataAdminDaoTest.java | 29 +-
sdks/java/io/hcatalog/build.gradle | 13 +-
sdks/java/io/iceberg/hive/build.gradle | 4 +-
sdks/java/io/iceberg/hive/exec/build.gradle | 8 +-
.../apache/beam/sdk/io/iceberg/RecordWriter.java | 77 +-
.../beam/sdk/io/iceberg/RecordWriterManager.java | 298 +++
.../sdk/io/iceberg/WriteGroupedRowsToFiles.java | 49 +-
.../sdk/io/iceberg/WriteUngroupedRowsToFiles.java | 164 +-
.../apache/beam/sdk/io/iceberg/IcebergIOIT.java | 285 +--
.../sdk/io/iceberg/RecordWriterManagerTest.java | 266 +++
.../beam/sdk/io/iceberg/TestDataWarehouse.java | 8 +-
.../beam/sdk/io/kafka/KafkaCommitOffset.java | 83 +-
.../beam/sdk/io/kafka/KafkaExactlyOnceSink.java | 16 +
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 82 +-
.../beam/sdk/io/kafka/KafkaUnboundedSource.java | 13 +
.../org/apache/beam/sdk/io/kafka/KafkaWriter.java | 18 +-
.../beam/sdk/io/kafka/ReadFromKafkaDoFn.java | 28 +-
.../beam/sdk/io/kafka/KafkaCommitOffsetTest.java | 169 +-
...KafkaIOReadImplementationCompatibilityTest.java | 23 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 73 +-
.../sdk/io/kafka/upgrade/KafkaIOTranslation.java | 28 +-
.../org/apache/beam/sdk/io/pulsar/PulsarIO.java | 15 +
.../beam/sdk/io/pulsar/ReadFromPulsarDoFn.java | 5 +
.../beam/sdk/io/pulsar/WriteToPulsarDoFn.java | 5 +
.../apache/beam/sdk/io/pulsar/package-info.java | 6 +-
.../solace/broker/SempBasicAuthClientExecutor.java | 23 +-
.../broker/SempBasicAuthClientExecutorTest.java | 81 +
sdks/python/apache_beam/io/aws/s3filesystem.py | 8 +
.../apache_beam/io/azure/blobstoragefilesystem.py | 8 +
sdks/python/apache_beam/io/filebasedsink.py | 1 +
sdks/python/apache_beam/io/filebasedsource.py | 2 +
sdks/python/apache_beam/io/filesystem.py | 8 +
sdks/python/apache_beam/io/filesystems.py | 13 +
sdks/python/apache_beam/io/gcp/bigquery_test.py | 11 +-
sdks/python/apache_beam/io/gcp/bigtableio.py | 7 +
sdks/python/apache_beam/io/gcp/bigtableio_test.py | 15 +
sdks/python/apache_beam/io/gcp/gcsfilesystem.py | 8 +
sdks/python/apache_beam/io/gcp/gcsio.py | 11 +-
sdks/python/apache_beam/io/gcp/pubsub.py | 67 +-
sdks/python/apache_beam/io/gcp/pubsub_test.py | 57 +
sdks/python/apache_beam/io/gcp/spanner_wrapper.py | 76 +
sdks/python/apache_beam/io/textio.py | 14 +-
sdks/python/apache_beam/io/textio_test.py | 30 +
sdks/python/apache_beam/metrics/metric.py | 14 +-
sdks/python/apache_beam/metrics/metric_test.py | 4 +-
sdks/python/apache_beam/ml/inference/base.py | 222 +-
sdks/python/apache_beam/ml/inference/base_test.py | 216 ++
.../python/apache_beam/options/pipeline_options.py | 8 +
sdks/python/apache_beam/pipeline.py | 15 +-
sdks/python/apache_beam/pipeline_test.py | 24 +
sdks/python/apache_beam/runners/common.py | 6 +
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../runners/interactive/cache_manager.py | 50 +-
.../runners/interactive/cache_manager_test.py | 7 +-
.../display/pcoll_visualization_test.py | 2 +-
.../apache-beam-jupyterlab-sidepanel/yarn.lock | 480 +++--
.../runners/interactive/interactive_beam.py | 108 +-
.../runners/interactive/interactive_runner_test.py | 46 +-
.../interactive/non_interactive_runner_test.py | 292 +++
.../runners/interactive/pipeline_fragment.py | 56 +-
.../runners/interactive/pipeline_fragment_test.py | 8 -
.../runners/interactive/recording_manager.py | 26 +-
.../runners/interactive/recording_manager_test.py | 2 +-
.../apache_beam/runners/interactive/utils.py | 3 +
.../runners/portability/prism_runner_test.py | 32 -
sdks/python/apache_beam/transforms/core.py | 57 +-
sdks/python/apache_beam/transforms/core_test.py | 71 +
.../apache_beam/transforms/error_handling.py | 126 ++
.../apache_beam/transforms/error_handling_test.py | 148 ++
.../transforms/external_transform_provider.py | 50 +-
.../external_transform_provider_it_test.py | 11 +-
sdks/python/apache_beam/transforms/util.py | 14 +
.../apache_beam/utils/multi_process_shared.py | 23 +
.../apache_beam/utils/multi_process_shared_test.py | 43 +
sdks/python/apache_beam/utils/subprocess_server.py | 3 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/integration_tests.py | 17 +
sdks/python/apache_beam/yaml/readme_test.py | 39 +-
sdks/python/apache_beam/yaml/standard_io.yaml | 31 +-
sdks/python/apache_beam/yaml/tests/spanner.yaml | 95 +
.../{version.py => yaml/tests/tsv.yaml} | 33 +-
sdks/python/apache_beam/yaml/yaml_combine.py | 2 +-
sdks/python/apache_beam/yaml/yaml_io.py | 12 +-
sdks/python/apache_beam/yaml/yaml_join.py | 3 +-
sdks/python/apache_beam/yaml/yaml_transform.py | 9 +-
.../apache_beam/yaml/yaml_transform_unit_test.py | 3 +-
.../container/base_image_requirements_manual.txt | 2 +-
.../container/py310/base_image_requirements.txt | 77 +-
.../container/py311/base_image_requirements.txt | 77 +-
.../container/py312/base_image_requirements.txt | 76 +-
.../container/py38/base_image_requirements.txt | 77 +-
.../container/py39/base_image_requirements.txt | 77 +-
sdks/python/gen_xlang_wrappers.py | 3 +-
sdks/python/setup.py | 30 +-
sdks/python/test-suites/portable/common.gradle | 5 +-
sdks/python/tox.ini | 15 +
sdks/standard_expansion_services.yaml | 4 +
sdks/typescript/package.json | 2 +-
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.58.0.md | 1 +
website/www/site/content/en/blog/beam-2.58.1.md | 45 +
website/www/site/content/en/blog/beam-2.59.0.md | 76 +
.../site/content/en/blog/unit-testing-in-beam.md | 198 ++
.../en/documentation/patterns/batch-elements.md | 45 +
.../content/en/documentation/patterns/overview.md | 5 +
.../en/documentation/patterns/shared-class.md | 93 +
.../content/en/documentation/programming-guide.md | 4 +-
.../content/en/documentation/sdks/yaml-combine.md | 6 +-
.../content/en/documentation/sdks/yaml-errors.md | 5 +-
.../en/documentation/sdks/yaml-inline-python.md | 2 -
.../content/en/documentation/sdks/yaml-join.md | 2 +-
.../www/site/content/en/documentation/sdks/yaml.md | 2 +-
.../www/site/content/en/get-started/downloads.md | 26 +-
.../partials/section-menu/en/documentation.html | 1 +
website/www/site/static/images/banner_desktop.png | Bin 79738 -> 216408 bytes
website/www/site/static/images/banner_mobile.png | Bin 64923 -> 581537 bytes
website/www/yarn.lock | 33 +-
346 files changed, 11075 insertions(+), 3873 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Python_Dependency.json => beam_PostCommit_Java_Examples_Dataflow_Java.json} (100%)
copy .github/trigger_files/{beam_PostCommit_Python_Dependency.json => beam_PreCommit_Python_ML.json} (100%)
create mode 100644 examples/multi-language/python/wordcount_external.py
create mode 100644 examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/schematransforms/ExtractWordsProvider.java
create mode 100644 examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/schematransforms/JavaCountProvider.java
create mode 100644 examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/schematransforms/WriteWordsProvider.java
create mode 100644 examples/notebooks/blog/unittests_in_beam.ipynb
copy sdks/java/io/google-ads/src/test/java/org/apache/beam/sdk/io/googleads/DummyRateLimitPolicy.java => runners/prism/java/src/main/java/org/apache/beam/runners/prism/PrismRunnerRegistrar.java (55%)
copy sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/package-info.java => runners/prism/java/src/main/java/org/apache/beam/runners/prism/TestPrismPipelineOptions.java (73%)
create mode 100644 runners/prism/java/src/main/java/org/apache/beam/runners/prism/TestPrismRunner.java
delete mode 100644 runners/prism/java/src/test/java/org/apache/beam/runners/prism/PrismPipelineResultTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextSourceTest.java
rename sdks/java/io/google-ads/src/main/java/org/apache/beam/sdk/io/googleads/{GoogleAdsV14.java => GoogleAdsV17.java} (89%)
rename sdks/java/io/google-ads/src/test/java/org/apache/beam/sdk/io/googleads/{GoogleAdsV14Test.java => GoogleAdsV17Test.java} (90%)
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerQuerySourceDef.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerReadSchemaTransformProvider.java
copy sdks/java/io/{pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/package-info.java => google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerSchemaRetrievalException.java} (69%)
copy sdks/java/io/{google-ads/src/main/java/org/apache/beam/sdk/io/googleads/GoogleAdsIO.java => google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerSourceDef.java} (60%)
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerTableSourceDef.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableSharedClientTest.java
create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriterManager.java
create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/RecordWriterManagerTest.java
create mode 100644 sdks/python/apache_beam/io/gcp/spanner_wrapper.py
create mode 100644 sdks/python/apache_beam/runners/interactive/non_interactive_runner_test.py
create mode 100644 sdks/python/apache_beam/transforms/error_handling.py
create mode 100644 sdks/python/apache_beam/transforms/error_handling_test.py
create mode 100644 sdks/python/apache_beam/yaml/tests/spanner.yaml
copy sdks/python/apache_beam/{version.py => yaml/tests/tsv.yaml} (50%)
create mode 100644 website/www/site/content/en/blog/beam-2.58.1.md
create mode 100644 website/www/site/content/en/blog/beam-2.59.0.md
create mode 100644 website/www/site/content/en/blog/unit-testing-in-beam.md
create mode 100644 website/www/site/content/en/documentation/patterns/batch-elements.md
create mode 100644 website/www/site/content/en/documentation/patterns/shared-class.md