You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2023/02/22 02:15:40 UTC
[beam] branch dependabot/go_modules/sdks/cloud.google.com/go/spanner-1.44.0 updated (81fd19270e4 -> 010cdc45bd2)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/spanner-1.44.0
in repository https://gitbox.apache.org/repos/asf/beam.git
discard 81fd19270e4 Bump cloud.google.com/go/spanner from 1.43.0 to 1.44.0 in /sdks
add 78c15648b51 Add Two Counter Metric in BigQuery Write Schema Transform (#25155)
add 8283e4fc0dd [prism] Add internal/config package (#25406)
add de16613c6d1 Fix typo in args (#25422)
add 983d9034b85 Added MetadataSpannerConfig class for generating SpannerConfig for accessing change stream metadata database (#25193)
add aeb0410d5e5 Remove ValueProvider from BigtableIO ReadChangeStream (#25409)
add 81f55cb1d48 Annotate Cloud Bigtable implementation details as Internal (#25403)
add ad5e17edb8f Add dependencies in some examples (#25425)
add feb248a41bc Add batching args to ModelHandlers docs (#25398)
add 38d33de8e22 Copy environment capabilities when creating the WorkerPool for Java pipelines
add bf3710c7045 Merge pull request #25418: Copy environment capabilities when creating the WorkerPool for Java pipelines
add aeab73e6a8e Update PythonMap transform to accept extra packages
add df2f70f547b Update the test
add d24bf68c6ef Address reviewer comments
add 6083f748a02 Merge pull request #25222: Update PythonMap transform to accept extra packages
add fafdf4a2121 Data sampling proto (#25421)
add 874bd458ffc Support ONNX runtime in RunInference API (#24911)
add be66a60124f Fix UpdateSchemaDestination breaking DynamicDestination in Bigquery BatchLoad (#25410)
add cff82b5d2ea Fix whitespace (#25432)
add e4fd7693796 [#25417][go] copy env details to dataflow images (#25431)
add cb5d5bb8fc3 Update jupyter-client requirement from <6.1.13,>=6.1.11 to >=6.1.11,<8.0.3 in /sdks/python (#25292)
add b64526a3a24 Add runtime metric to TFT tests (#25242)
add 783584d0773 Fix typo tranform; workaround non-ascii char (#25428)
add 9b77bf9b130 Add support for loading torchscript models (#25321)
add 198b93eacef [#24971] Add a retry policy for JmsIO #24971 (#24973)
add 6f7d2fbaeb4 (#25316) Enable LZMA compression in Python SDK I/O (#25317)
add bb5e200df70 Add WatchFilePattern (#25393)
add 0a0bff5a049 Support gauge metrics in portable mode (#25396)
add 007713c8320 Validate that GBK coders are always set correctly. (#25394)
add a1e07773e8b Ignore flags for beam_sql magic
add 03982ea4583 Merge remote-tracking branch 'origin/master'
add 2c3fd4ff4ba Raise the error when some functions are misused
add 154ff573837 change how to import
add 15699670853 Raise the RuntimeError message when some InteractiveRunner functions are misused
add 849692ef7b6 Fix tox error running hdfsIntegrationTest (#25446)
add 299be58cf99 Adding support for DLQ for ZetaSQL (#25426)
add 1dfd89dffc9 Remove python 3.6 references (#25445)
add 5b0b5541d43 Fix pulling licenses (#25456)
add d232795556a Update grafana dashboard to handle name change for CombinerTableBenchmark to PrecombineGroupingTableBenchmark (#25452)
add ed7eae82b31 Test loading URLs (#25034)
add 67b7962f8e7 Fixing issue with ErrorCapture transform where pipeline issues are caused by lack of proper expansion (#25465)
add e1e5cdf72fe [24464] Finalize FileWriteSchemaTransformProvider (#25420)
add bf6d24223f7 Use codecov-action@v3; v2 is no longer supported. (#25477)
add 3e080ff212d Update the title of the wordcount quickstart (#25471)
add 52b604076b8 [Go SDK] add retries to connect with expansion service (#25237)
add 7fa3cd387a0 SpannerIO: parameterizing partitionQuery timeout (#25236)
add bd3511571ff Use example id as CloudPath (#25487)
add fd2b16139fd SpannerIO: Handling pg spanner.commit_timestamp (#25479)
add 26735eb9007 Fix provider to be found by AutoService (#25491)
add 8bf324d7826 [Python] Added Tensorflow Model Handler (#25368)
add a64aa0f57ba [#24789][prism] internal/coders.go and tests (#25476)
add 286100eb389 Update documents for 2.45 (#25407)
add 65cc0e6ed2f Update documents for 2.45 (#25500)
add d8fbbf03a79 Update python container images (#25475)
add fce8b6555ce Fix beam.Row.__eq__ for rows with trailing columns (#23876)
add 3edf78996cd TFMA notebook showing ExtractEvaluateAndWriteResult Tranfsorm (#25381)
add 9e737d38f84 Fix whitespace check (#25514)
add 6e08092bd63 Add back links removed in #24744 (#25513)
add 2aa77521413 Fix pull_licenses_java returning 404 from opensource.org
add 1b1cc449ef8 Merge pull request #25515: Fix pull_licenses_java returning 404 from opensource.org
add 9fb9d2cb415 [#24789][prism] internal/jobservices metrics + tests (#25497)
add 29ea6e0eb8a [Go SDK]: Allow SDF methods to have context param and error return value (#25437)
add 5fc19fffcdf Move closing milestone out of PMC-only tasks (#25516)
add a82084ce87b update GCP cloud libraries BOM to 26.8.0 (#25470)
add c210df3d3b3 Fix interface{} in iter& emit type of DoFn in Go (#25203)
add 545a2e63ca7 Task #25064: Python SDK Data sampling implementation (#25093)
add 4e6d751f92e Add support for all Java based portable runners to consume elements embedding on the control response. (#25463)
add 684917f9bc5 Swap Java SDK container to use eclipse-temurin as the base instead of OpenJDK (#25402)
add d391bf2b3a5 Update BigQueryIO documentation with details on how to override the query execution project (#25400)
add 40838f76447 Support Avro GenericRecord as a valid format for StorageWrite API on BigQueryIO (#24274)
add ee9244d6261 complete examples links fixed
add 20c8d3a2ba1 Merge pull request #25527: [CdapIO][SparkReceiverIO] Complete examples links in READMEs and CdapIO website page fixed
add 6de67443528 Stop use of self hosted runners for some workflows. (#25542)
add 29856ae9471 Run prbot updates on hosted runners (#25544)
add f358fcdc8c8 prism-fixstatic (#25546)
add 86920261ca7 Update Go SDK minimum Go version to 1.19 (#25545)
add 2b6291250f9 Replace more uses of `ClassLoadingStrategy.Default.INJECTION` (#23210)
add 907533220b8 Use WindowedValue.withValue on hot paths #21250 (#25519)
add de7eb2d83b0 Stop paying the iterator object creation tax in MultiplexingMetricTrackingFnDataReceiver (#25540)
add 977e53104f4 [prism] add windowing strategy (#25518)
add c160a081923 Fix Tensorflow intergration test model path (#25553)
add d652d054ecd [Go SDK]: Retrieve file size in CreateInitialRestriction in textio.Read (#25535)
add ca1ec257c4a [#24789][prism] internal/worker + tentative data (#25478)
add 6667eb4741b [#24789][prism] add preprocessor and test (#25520)
add 6e53f13871a Bump github.com/tetratelabs/wazero from 1.0.0-pre.7 to 1.0.0-pre.9 in /sdks (#25560)
add bc0284b5cae [#24789][prism] Add testdofns for later (#25557)
add fc4b80c091c [#24789][prism] Minimum required jobservices (#25556)
add 85ebc2f46b0 [#24789][prism] Handlers for combine, ParDo, GBK, Flatten (#25558)
add 63e4c37cded [prism] Tiny lint fixes (#25564)
add 108d097bbe9 minor error handling fixes (#25555)
add 921bc7b9470 [prism] Add in element manager (#25565)
add 903fbf9e721 Better error for missing timezone info (#25444)
add b92600adafe Revert "Replace more uses of `ClassLoadingStrategy.Default.INJECTION` (#23210)" (#25566)
add e80596f5d41 Use context param and error return value in mongodbio.Read SDF methods (#25536)
add af2a86139cf Revert workers for java tests (#25579)
add 3bfc3136a2a remove unnecessary error check (#25576)
add 3a6259900e6 Making the default BigQuery write disposition to be WRITE_APPEND (#25434)
add 33750c1dabf [Python] Support loading of TF models with saved weights (#25496)
add 5ace2ec4caf Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#25572)
add 010cdc45bd2 Bump cloud.google.com/go/spanner from 1.43.0 to 1.44.0 in /sdks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (81fd19270e4)
\
N -- N -- N refs/heads/dependabot/go_modules/sdks/cloud.google.com/go/spanner-1.44.0 (010cdc45bd2)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../actions/setup-self-hosted-action/action.yml | 8 +-
.github/workflows/build_playground_frontend.yml | 2 +-
.github/workflows/git_tag_released_version.yml | 2 +-
.github/workflows/go_tests.yml | 6 +-
.github/workflows/issue-tagger.yml | 2 +-
.github/workflows/java_tests.yml | 8 +-
.github/workflows/label_prs.yml | 2 +-
.github/workflows/playground_deploy_backend.yml | 2 +-
.github/workflows/playground_frontend_test.yml | 2 +-
.github/workflows/pr-bot-new-prs.yml | 2 +-
.github/workflows/pr-bot-pr-updates.yml | 2 +-
.github/workflows/pr-bot-prs-needing-attention.yml | 2 +-
.github/workflows/publish_github_release_notes.yml | 4 +-
.github/workflows/python_tests.yml | 2 +-
.github/workflows/reportGenerator.yml | 2 +-
.github/workflows/self-assign.yml | 2 +-
.github/workflows/stale.yml | 2 +-
.github/workflows/triaged-on-assign.yml | 2 +-
.../job_CloudMLBenchmarkTests_Python.groovy | 13 +
.../perftests_metrics/Java_JMH_benchmarks.json | 12 +-
.test-infra/tools/README.md | 4 +-
.test-infra/tools/python_installer.sh | 2 +-
CHANGES.md | 23 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 12 +-
dev-support/docker/Dockerfile | 2 +-
examples/notebooks/beam-ml/tfma_beam.ipynb | 742 +++++++++++++++++++
learning/tour-of-beam/frontend/pubspec.yaml | 2 +-
.../beam/model/fn_execution/v1/beam_fn_api.proto | 14 +-
playground/README.md | 6 +
playground/backend/cmd/server/controller.go | 46 +-
playground/backend/cmd/server/controller_test.go | 22 +-
.../internal/components/cache_component_test.go | 2 +-
.../internal/db/datastore/datastore_db_test.go | 4 +-
.../db/mapper/precompiled_object_mapper.go | 2 +-
.../db/mapper/precompiled_object_mapper_test.go | 20 +-
.../backend/internal/utils/datastore_utils.go | 11 -
.../backend/internal/utils/datastore_utils_test.go | 42 --
playground/frontend/Dockerfile | 3 +-
playground/frontend/README.md | 164 ++++-
playground/frontend/build.gradle | 14 +
.../common/examples.dart} | 18 +-
.../common/widget_tester.dart} | 17 +-
.../integration_test/initial_urls_test.dart | 272 +++++++
.../standalone_change_example_sdk_run_test.dart | 66 +-
.../standalone_default_examples_test.dart | 95 +++
playground/frontend/lib/locator.dart | 9 +-
playground/frontend/lib/main.dart | 5 +-
.../no_url_example_loading_descriptor.dart | 3 +
.../widgets/embedded_actions.dart | 4 +-
.../lib/src/assets/assets.gen.dart | 22 +-
.../lib/src/cache/example_cache.dart | 7 +-
.../lib/src/constants/constants.dart} | 5 +-
.../example_loaders/content_example_loader.dart | 1 +
.../example_loaders/http_example_loader.dart | 4 +
.../example_loaders/standard_example_loader.dart | 2 +-
.../user_shared_example_loader.dart | 5 +-
.../snippet_file_editing_controller.dart | 6 +-
...catalog_default_example_loading_descriptor.dart | 12 +-
.../content_example_loading_descriptor.dart | 36 +-
.../empty_example_loading_descriptor.dart | 5 +-
.../example_loading_descriptor.dart | 2 +
.../examples_loading_descriptor.dart | 20 +
.../http_example_loading_descriptor.dart | 8 +
.../standard_example_loading_descriptor.dart | 8 +
.../user_shared_example_loading_descriptor.dart | 8 +
.../lib/src/models/example_view_options.dart | 9 +
.../frontend/playground_components/pubspec.yaml | 16 +-
.../test/src/common/example_cache.mocks.dart | 94 ++-
.../src/common/example_repository_mock.mocks.dart | 62 +-
.../src/controllers/example_loaders/common.dart | 3 +
.../example_loader_factory_test.dart | 3 +
.../examples_loader_test.mocks.dart | 248 +++++--
.../http_example_loader_test.mocks.dart | 94 ++-
.../playground_controller_test.mocks.dart | 119 ++-
...og_default_example_loading_descriptor_test.dart | 17 +-
.../content_example_loading_descriptor_test.dart | 16 +-
.../empty_example_loading_descriptor_test.dart | 4 +
.../examples_loading_descriptor_test.dart | 36 +
.../http_example_loading_descriptor_test.dart | 14 +-
.../standard_example_loading_descriptor_test.dart | 16 +-
...ser_shared_example_loading_descriptor_test.dart | 14 +-
.../repositories/code_repository_test.mocks.dart | 115 ++-
.../example_repository_test.mocks.dart | 156 +++-
.../lib/playground_components_dev.dart | 19 +-
.../playground_components_dev/lib/src/code.dart | 3 +-
.../lib/src/examples.dart | 35 -
.../lib/src/examples/example_descriptor.dart | 131 ++++
.../lib/src/examples/go/content/example.go} | 28 +-
.../lib/src/examples/go/example.dart} | 61 +-
.../lib/src/examples/go/minimal_word_count.dart | 61 ++
.../lib/src/examples/go/word_count.dart} | 17 +-
.../java/aggregation_max.dart} | 23 +-
.../lib/src/examples/java/minimal_word_count.dart} | 22 +-
.../lib/src/examples/python/aggregation_mean.dart} | 18 +-
.../examples/python/word_count_with_metrics.dart} | 21 +-
.../scio/common.dart} | 11 +-
.../lib/src/examples/scio/minimal_word_count.dart} | 23 +-
.../playground_components_dev/lib/src/expect.dart | 43 +-
.../lib/src/widget_tester.dart | 51 ++
.../playground_components_dev/pubspec.yaml | 8 +-
playground/frontend/pubspec.lock | 808 ++++++++++++++-------
playground/frontend/pubspec.yaml | 12 +-
.../states/example_selector_state_test.mocks.dart | 56 +-
.../runners/core/metrics/MetricsContainerImpl.java | 47 +-
.../core/metrics/MonitoringInfoConstants.java | 2 +
.../core/metrics/MetricsContainerImplTest.java | 35 +
.../dataflow/BatchStatefulParDoOverrides.java | 2 +-
.../beam/runners/dataflow/DataflowRunner.java | 20 +-
.../options/DataflowPipelineDebugOptions.java | 4 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 5 +-
.../control/DefaultJobBundleFactory.java | 3 +
.../fnexecution/control/SdkHarnessClient.java | 4 +
.../environment/EmbeddedEnvironmentFactory.java | 7 +-
.../runners/samza/SamzaExecutionEnvironment.java | 2 +-
sdks/go.mod | 29 +-
sdks/go.sum | 39 +-
sdks/go/examples/pingpong/pingpong.go | 2 +-
sdks/go/examples/wasm/README.md | 6 +-
sdks/go/examples/wasm/wasm.go | 6 +-
sdks/go/pkg/beam/core/funcx/output.go | 5 +-
sdks/go/pkg/beam/core/funcx/output_test.go | 2 +
sdks/go/pkg/beam/core/funcx/sideinput.go | 3 +
sdks/go/pkg/beam/core/funcx/sideinput_test.go | 1 +
sdks/go/pkg/beam/core/graph/fn.go | 110 ++-
sdks/go/pkg/beam/core/graph/fn_test.go | 87 +++
sdks/go/pkg/beam/core/runtime/exec/datasink.go | 2 +-
sdks/go/pkg/beam/core/runtime/exec/datasource.go | 8 +-
.../pkg/beam/core/runtime/exec/datasource_test.go | 12 +-
sdks/go/pkg/beam/core/runtime/exec/sdf.go | 134 +++-
sdks/go/pkg/beam/core/runtime/exec/sdf_invokers.go | 339 ++++-----
.../beam/core/runtime/exec/sdf_invokers_arity.go | 336 +++++++++
.../beam/core/runtime/exec/sdf_invokers_arity.tmpl | 246 +++++++
.../beam/core/runtime/exec/sdf_invokers_test.go | 250 ++++++-
sdks/go/pkg/beam/core/runtime/exec/sdf_test.go | 17 +-
sdks/go/pkg/beam/core/runtime/pipelinex/util.go | 3 +-
sdks/go/pkg/beam/core/runtime/xlangx/expand.go | 31 +-
sdks/go/pkg/beam/io/mongodbio/read.go | 29 +-
sdks/go/pkg/beam/io/parquetio/parquetio.go | 4 +-
sdks/go/pkg/beam/io/textio/textio.go | 39 +-
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 45 +-
.../beam/runners/dataflow/dataflowlib/job_test.go | 86 +++
sdks/go/pkg/beam/runners/prism/internal/coders.go | 243 +++++++
.../pkg/beam/runners/prism/internal/coders_test.go | 377 ++++++++++
.../beam/runners/prism/internal/config/config.go | 247 +++++++
.../runners/prism/internal/config/config_test.go | 221 ++++++
.../internal/{urns/urns_test.go => engine/data.go} | 30 +-
.../prism/internal/engine/elementmanager.go | 675 +++++++++++++++++
.../prism/internal/engine/elementmanager_test.go | 516 +++++++++++++
.../{urns/urns_test.go => engine/strategy.go} | 46 +-
.../prism/internal/engine/strategy_test.go} | 33 +-
.../internal/{urns/urns_test.go => execute.go} | 30 +-
.../beam/runners/prism/internal/handlecombine.go | 209 ++++++
.../pkg/beam/runners/prism/internal/handlepardo.go | 244 +++++++
.../beam/runners/prism/internal/handlerunner.go | 298 ++++++++
.../runners/prism/internal/jobservices/artifact.go | 81 +++
.../beam/runners/prism/internal/jobservices/job.go | 120 +++
.../prism/internal/jobservices/management.go | 142 ++++
.../runners/prism/internal/jobservices/metrics.go | 495 +++++++++++++
.../prism/internal/jobservices/metrics_test.go | 134 ++++
.../runners/prism/internal/jobservices/server.go | 82 +++
.../prism/internal/jobservices/server_test.go | 79 ++
.../pkg/beam/runners/prism/internal/preprocess.go | 148 ++++
.../beam/runners/prism/internal/preprocess_test.go | 181 +++++
.../pkg/beam/runners/prism/internal/testdofns.go | 349 +++++++++
.../beam/runners/prism/internal/testdofns_test.go | 55 ++
.../pkg/beam/runners/prism/internal/urns/urns.go | 2 +-
.../beam/runners/prism/internal/urns/urns_test.go | 2 +-
.../beam/runners/prism/internal/worker/bundle.go | 114 +++
.../runners/prism/internal/worker/bundle_test.go | 52 ++
.../beam/runners/prism/internal/worker/worker.go | 421 +++++++++++
.../runners/prism/internal/worker/worker_test.go | 281 +++++++
.../test/integration/primitives/checkpointing.go | 2 +-
sdks/java/container/Dockerfile | 2 +-
.../container/license_scripts/dep_urls_java.yaml | 6 +-
.../license_scripts/pull_licenses_java.py | 4 +
.../apache/beam/sdk/coders/RowCoderGenerator.java | 12 +-
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 20 +-
.../extensions/avro/schemas/utils/AvroUtils.java | 14 +-
.../extensions/python/transforms/PythonMap.java | 34 +-
.../extensions/python/transforms/RunInference.java | 8 +-
.../SqlTransformSchemaTransformProvider.java | 9 +
.../extensions/sql/zetasql/BeamZetaSqlCalcRel.java | 112 ++-
.../sql/zetasql/BeamZetaSqlCalcRelTest.java | 70 ++
.../sql/zetasql/ZetaSqlDialectSpecTest.java | 14 +-
.../fn/harness/jmh/ProcessBundleBenchmark.java | 16 +-
.../fn/harness/jmh/ProcessBundleBenchmarkTest.java | 18 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 257 +++++--
.../harness/data/PCollectionConsumerRegistry.java | 10 +-
sdks/java/io/cdap/README.md | 2 +-
sdks/java/io/csv/build.gradle | 1 +
.../java/org/apache/beam/sdk/io/csv/CsvIO.java | 50 +-
.../org/apache/beam/sdk/io/csv/CsvIOTestData.java | 12 +-
.../apache/beam/sdk/io/csv/CsvIOTestJavaBeans.java | 265 -------
.../beam/sdk/io/csv/CsvIOWriteDisplayDataTest.java | 76 ++
.../org/apache/beam/sdk/io/csv/CsvIOWriteTest.java | 6 +-
.../beam/sdk/io/csv/RowToCsvCSVFormatTest.java | 6 +-
.../io/csv/RowToCsvPredefinedCSVFormatsTest.java | 6 +-
sdks/java/io/file-schema-transform/build.gradle | 1 +
.../CsvWriteSchemaTransformFormatProvider.java | 65 +-
.../FileWriteSchemaTransformConfiguration.java | 19 +-
.../FileWriteSchemaTransformFormatProviders.java | 4 +-
.../FileWriteSchemaTransformProvider.java | 3 +
...FileWriteSchemaTransformFormatProviderTest.java | 248 +++++++
...FileWriteSchemaTransformFormatProviderTest.java | 10 +-
.../AvroGenericRecordToStorageApiProto.java | 378 ++++++++++
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 48 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 52 +-
.../io/gcp/bigquery/BigQueryResourceNaming.java | 1 +
.../beam/sdk/io/gcp/bigquery/RowWriterFactory.java | 4 +
...StorageApiDynamicDestinationsGenericRecord.java | 95 +++
.../io/gcp/bigquery/UpdateSchemaDestination.java | 76 +-
...ueryStorageWriteApiSchemaTransformProvider.java | 45 +-
.../beam/sdk/io/gcp/bigtable/BigtableConfig.java | 2 +
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 63 +-
.../changestreams/ByteStringRangeHelper.java | 2 +
.../changestreams/ChangeStreamMetrics.java | 2 +
.../changestreams/ChangeStreamMutation.java | 3 +
.../bigtable/changestreams/TimestampConverter.java | 2 +
.../bigtable/changestreams/UniqueIdGenerator.java | 2 +
.../changestreams/action/ActionFactory.java | 2 +
.../changestreams/action/ChangeStreamAction.java | 2 +
.../action/DetectNewPartitionsAction.java | 2 +
.../action/GenerateInitialPartitionsAction.java | 2 +
.../action/ReadChangeStreamPartitionAction.java | 2 +
.../changestreams/dao/ChangeStreamDao.java | 2 +
.../gcp/bigtable/changestreams/dao/DaoFactory.java | 2 +
.../changestreams/dao/MetadataTableAdminDao.java | 2 +
.../changestreams/dao/MetadataTableDao.java | 2 +
.../dofn/DetectNewPartitionsDoFn.java | 2 +
.../changestreams/dofn/InitializeDoFn.java | 2 +
.../dofn/ReadChangeStreamPartitionDoFn.java | 2 +
.../changestreams/model/PartitionRecord.java | 2 +
.../ReadChangeStreamPartitionProgressTracker.java | 2 +
.../changestreams/restriction/StreamProgress.java | 2 +
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 8 +-
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 22 +
.../beam/sdk/io/gcp/spanner/SpannerConfig.java | 28 +
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 10 +-
.../beam/sdk/io/gcp/spanner/SpannerSchema.java | 3 +
.../MetadataSpannerConfigFactory.java | 118 +++
.../gcp/spanner/changestreams/dao/DaoFactory.java | 2 +-
.../AvroGenericRecordToStorageApiProtoTest.java | 473 ++++++++++++
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 109 ++-
.../bigquery/BigQuerySchemaUpdateOptionsIT.java | 176 +++--
...StorageWriteApiSchemaTransformProviderTest.java | 169 ++++-
.../beam/sdk/io/gcp/spanner/SpannerReadIT.java | 80 ++
.../beam/sdk/io/gcp/spanner/SpannerSchemaTest.java | 8 +-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 257 +++++--
.../apache/beam/sdk/io/jms/RetryConfiguration.java | 71 ++
.../java/org/apache/beam/sdk/io/jms/JmsIOTest.java | 229 +++++-
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 2 +-
.../org/apache/beam/sdk/io/kinesis/KinesisIO.java | 2 +-
sdks/java/io/sparkreceiver/2/README.md | 2 +-
.../apache/beam/sdk/tpcds/SqlTransformRunner.java | 2 +-
.../apache_beam/examples/inference/README.md | 202 ++++++
.../inference/onnx_sentiment_classification.py | 147 ++++
...ytorch_image_classification_with_side_inputs.py | 218 ++++++
.../inference/tensorflow_imagenet_segmentation.py | 128 ++++
.../inference/tensorflow_mnist_classification.py | 118 +++
.../inference/tensorflow_mnist_with_weights.py | 93 +++
sdks/python/apache_beam/io/fileio.py | 6 +-
sdks/python/apache_beam/io/filesystem.py | 21 +-
sdks/python/apache_beam/io/filesystem_test.py | 25 +-
.../apache_beam/io/gcp/bigquery_file_loads.py | 2 +-
.../apache_beam/ml/gcp/recommendations_ai.py | 2 +-
sdks/python/apache_beam/ml/inference/base_test.py | 22 +-
.../apache_beam/ml/inference/onnx_inference.py | 132 ++++
.../ml/inference/onnx_inference_it_test.py | 94 +++
.../ml/inference/onnx_inference_test.py | 470 ++++++++++++
.../apache_beam/ml/inference/pytorch_inference.py | 149 +++-
.../ml/inference/pytorch_inference_test.py | 131 ++++
.../apache_beam/ml/inference/sklearn_inference.py | 13 +
.../ml/inference/tensorflow_inference.py | 275 +++++++
.../ml/inference/tensorflow_inference_it_test.py | 145 ++++
.../ml/inference/tensorflow_inference_test.py | 146 ++++
.../ml/inference/tensorflow_tests_requirements.txt | 22 +-
sdks/python/apache_beam/ml/inference/utils.py | 116 +++
sdks/python/apache_beam/ml/inference/utils_test.py | 103 +++
sdks/python/apache_beam/pvalue.py | 7 +-
sdks/python/apache_beam/pvalue_test.py | 18 +
sdks/python/apache_beam/runners/common.py | 44 ++
.../runners/dataflow/internal/apiclient.py | 4 +-
.../runners/dataflow/internal/apiclient_test.py | 16 +-
.../runners/direct/transform_evaluator.py | 5 +-
.../runners/interactive/cache_manager.py | 2 +-
.../runners/interactive/pipeline_fragment.py | 5 +
.../runners/interactive/pipeline_fragment_test.py | 8 +
.../runners/interactive/pipeline_instrument.py | 2 +-
.../python/apache_beam/runners/pipeline_context.py | 12 +-
.../runners/portability/fn_api_runner/fn_runner.py | 2 +
.../runners/portability/portable_runner.py | 7 +
.../apache_beam/runners/worker/bundle_processor.py | 95 ++-
.../runners/worker/bundle_processor_test.py | 132 ++++
.../apache_beam/runners/worker/data_sampler.py | 156 ++++
.../runners/worker/data_sampler_test.py | 192 +++++
.../apache_beam/runners/worker/sdk_worker.py | 34 +-
.../apache_beam/runners/worker/sdk_worker_main.py | 9 +-
.../apache_beam/runners/worker/sdk_worker_test.py | 55 +-
.../benchmarks/chicago_taxi/trainer/model.py | 2 +-
.../benchmarks/cloudml/cloudml_benchmark_test.py | 61 ++
.../benchmarks/cloudml/criteo_tft/criteo.py | 2 +-
sdks/python/apache_beam/transforms/environments.py | 1 +
sdks/python/apache_beam/typehints/schemas.py | 22 +-
sdks/python/apache_beam/utils/timestamp.py | 5 +
sdks/python/apache_beam/utils/timestamp_test.py | 5 +-
.../container/py310/base_image_requirements.txt | 46 +-
.../container/py37/base_image_requirements.txt | 30 +-
.../container/py38/base_image_requirements.txt | 32 +-
.../container/py39/base_image_requirements.txt | 30 +-
sdks/python/pytest.ini | 2 +
sdks/python/setup.py | 2 +-
sdks/python/test-suites/dataflow/common.gradle | 8 +-
sdks/python/test-suites/direct/common.gradle | 30 +-
sdks/python/test-suites/tox/py38/build.gradle | 17 +
sdks/python/tox.ini | 37 +
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.45.0.md | 220 ++++++
.../site/content/en/contribute/release-guide.md | 8 +-
.../content/en/documentation/io/built-in/cdap.md | 2 +-
.../documentation/io/built-in/google-bigquery.md | 5 +
.../site/content/en/documentation/io/connectors.md | 4 +-
.../en/documentation/ml/tensorrt-runinference.md | 2 +-
.../documentation/pipelines/test-your-pipeline.md | 12 +
.../content/en/documentation/programming-guide.md | 2 +-
.../www/site/content/en/get-started/downloads.md | 14 +-
.../site/content/en/get-started/quickstart-go.md | 2 +-
.../site/content/en/get-started/quickstart-py.md | 4 +-
website/www/site/data/authors.yml | 3 +
.../www/site/layouts/shortcodes/playground.html | 2 +
.../layouts/shortcodes/playground_snippet.html | 5 +-
330 files changed, 18124 insertions(+), 2290 deletions(-)
create mode 100755 examples/notebooks/beam-ml/tfma_beam.ipynb
copy playground/frontend/{lib/modules/examples/models/example_loading_descriptors/no_url_example_loading_descriptor.dart => integration_test/common/examples.dart} (79%)
copy playground/frontend/{playground_components/test/src/models/example_loading_descriptors/empty_example_loading_descriptor_test.dart => integration_test/common/widget_tester.dart} (74%)
create mode 100644 playground/frontend/integration_test/initial_urls_test.dart
create mode 100644 playground/frontend/integration_test/standalone_default_examples_test.dart
rename playground/frontend/{playground_components_dev/lib/src/example_names.dart => playground_components/lib/src/constants/constants.dart} (86%)
delete mode 100644 playground/frontend/playground_components_dev/lib/src/examples.dart
create mode 100644 playground/frontend/playground_components_dev/lib/src/examples/example_descriptor.dart
copy playground/frontend/{playground_components/lib/src/models/example_loading_descriptors/example_loading_descriptor.dart => playground_components_dev/lib/src/examples/go/content/example.go} (65%)
copy playground/frontend/{playground_components/test/src/models/example_loading_descriptors/http_example_loading_descriptor_test.dart => playground_components_dev/lib/src/examples/go/example.dart} (55%)
create mode 100644 playground/frontend/playground_components_dev/lib/src/examples/go/minimal_word_count.dart
copy playground/frontend/{lib/modules/examples/models/example_loading_descriptors/no_url_example_loading_descriptor.dart => playground_components_dev/lib/src/examples/go/word_count.dart} (73%)
rename playground/frontend/playground_components_dev/lib/src/{example_paths.dart => examples/java/aggregation_max.dart} (68%)
copy playground/frontend/{playground_components/test/src/models/example_loading_descriptors/empty_example_loading_descriptor_test.dart => playground_components_dev/lib/src/examples/java/minimal_word_count.dart} (70%)
copy playground/frontend/{lib/modules/examples/models/example_loading_descriptors/no_url_example_loading_descriptor.dart => playground_components_dev/lib/src/examples/python/aggregation_mean.dart} (71%)
copy playground/frontend/{playground_components/test/src/models/example_loading_descriptors/empty_example_loading_descriptor_test.dart => playground_components_dev/lib/src/examples/python/word_count_with_metrics.dart} (70%)
rename playground/frontend/playground_components_dev/lib/src/{example_outputs.dart => examples/scio/common.dart} (75%)
copy playground/frontend/{playground_components/test/src/models/example_loading_descriptors/empty_example_loading_descriptor_test.dart => playground_components_dev/lib/src/examples/scio/minimal_word_count.dart} (67%)
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/sdf_invokers_arity.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/sdf_invokers_arity.tmpl
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/coders.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/coders_test.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/config/config.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/config/config_test.go
copy sdks/go/pkg/beam/runners/prism/internal/{urns/urns_test.go => engine/data.go} (55%)
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager_test.go
copy sdks/go/pkg/beam/runners/prism/internal/{urns/urns_test.go => engine/strategy.go} (54%)
copy sdks/go/pkg/beam/{core/funcx/output_test.go => runners/prism/internal/engine/strategy_test.go} (52%)
copy sdks/go/pkg/beam/runners/prism/internal/{urns/urns_test.go => execute.go} (60%)
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/handlecombine.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/handlepardo.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/handlerunner.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/jobservices/artifact.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/jobservices/job.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/jobservices/management.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/jobservices/metrics.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/jobservices/metrics_test.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/jobservices/server.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/jobservices/server_test.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/preprocess.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/preprocess_test.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/testdofns.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/testdofns_test.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/worker/bundle.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/worker/bundle_test.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/worker/worker.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/worker/worker_test.go
delete mode 100644 sdks/java/io/csv/src/test/java/org/apache/beam/sdk/io/csv/CsvIOTestJavaBeans.java
create mode 100644 sdks/java/io/csv/src/test/java/org/apache/beam/sdk/io/csv/CsvIOWriteDisplayDataTest.java
create mode 100644 sdks/java/io/file-schema-transform/src/test/java/org/apache/beam/sdk/io/fileschematransform/CsvFileWriteSchemaTransformFormatProviderTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroGenericRecordToStorageApiProto.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiDynamicDestinationsGenericRecord.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/MetadataSpannerConfigFactory.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/AvroGenericRecordToStorageApiProtoTest.java
create mode 100644 sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/RetryConfiguration.java
create mode 100644 sdks/python/apache_beam/examples/inference/onnx_sentiment_classification.py
create mode 100644 sdks/python/apache_beam/examples/inference/pytorch_image_classification_with_side_inputs.py
create mode 100644 sdks/python/apache_beam/examples/inference/tensorflow_imagenet_segmentation.py
create mode 100644 sdks/python/apache_beam/examples/inference/tensorflow_mnist_classification.py
create mode 100644 sdks/python/apache_beam/examples/inference/tensorflow_mnist_with_weights.py
create mode 100644 sdks/python/apache_beam/ml/inference/onnx_inference.py
create mode 100644 sdks/python/apache_beam/ml/inference/onnx_inference_it_test.py
create mode 100644 sdks/python/apache_beam/ml/inference/onnx_inference_test.py
create mode 100644 sdks/python/apache_beam/ml/inference/tensorflow_inference.py
create mode 100644 sdks/python/apache_beam/ml/inference/tensorflow_inference_it_test.py
create mode 100644 sdks/python/apache_beam/ml/inference/tensorflow_inference_test.py
copy .github/workflows/issue-tagger.yml => sdks/python/apache_beam/ml/inference/tensorflow_tests_requirements.txt (63%)
create mode 100644 sdks/python/apache_beam/ml/inference/utils_test.py
create mode 100644 sdks/python/apache_beam/runners/worker/data_sampler.py
create mode 100644 sdks/python/apache_beam/runners/worker/data_sampler_test.py
create mode 100644 website/www/site/content/en/blog/beam-2.45.0.md