You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by an...@apache.org on 2023/12/19 16:59:14 UTC
(beam) branch compute_cost updated (db36ffbbadb -> a985c24ba84)
This is an automated email from the ASF dual-hosted git repository.
anandinguva pushed a change to branch compute_cost
in repository https://gitbox.apache.org/repos/asf/beam.git
omit db36ffbbadb Merge remote-tracking branch 'origin/master' into compute_cost
omit 016573bf4f1 Add computing approximate cost for dataflow jobs
omit cdaa3badd00 Fix lint
omit 8fb0ca2d288 Append job id to each metric
omit c14893cba87 Publish job (#29685)
add eff6c203991 Bump actions/setup-java from 3 to 4 (#29573)
add 4431479fd90 Add InfluxDB credentials as env variables where they are missing
add 8ecfaa64fa8 Merge pull request #29716:Add InfluxDB credentials as env variables
add f9479c68190 Support BigQuery tables with spaces (#29649)
add d24c40a125c Bump com.gradle.common-custom-user-data-gradle-plugin (#29632)
add 5e00feea6c6 Update links to point to About Beam ML (#29709)
add 12a4f677c69 Fixes dictionary size change in direct runners (#29122)
add 649da779acd Update to most recent dataflow python container (#29717)
add f4d4a0d4696 Add utility method to parse labels/base name from a mangled bigquery metric name (#29500)
add 0e85810d0d6 Add HistogramData::getAndReset method (#29484)
add 475a0c78690 Fix syntax (#29719)
add efe7e6aad1f Deal with trailing slash in tempRoot (#29478)
add d59e1920d23 Bump dataflowLegacyContainerVersion (#29723)
add 90e79ae373a [Go SDK]: Implement natsio.Read transform for reading from NATS (#29410)
add 276aa023903 Add suffix for vocab files in tft transforms (#29720)
add 8be85d29f53 fix: optimize segment reader (#29694)
add b2073c6c986 Fix typo in notebook (#29728)
add c8deda71743 Upgrade go version to resolve vulnerabilities (#29743)
add 5d42cb654c0 revert kafka host change (#29725)
add 86720287c21 Deep-copy the process bundle descriptor when creating a new bundle processor. (#29742)
add 06c3340e338 Add logs to confirm JvmInitializer completed running and Spanner successfully connected. (#29727)
add 8f90fdd62d3 Upgrade Debian Bullseye to Bookworm (#29721)
add 951b3b1a81b Per DoFn latency instrumentation (#29592)
add fa47640888c Remove comment body from action (#29750)
add ad2d8dc6e30 Update Bigtable python client version (#29753)
add e37fa77c3b6 Pass getPort method through JDBC Resource Managers (#29732)
add 8d41b62e828 Moving to 2.54.0-SNAPSHOT on master branch.
add c49e9ec7dc7 test revert _beam_schema_proto (#29754)
add db0bb446aa0 Add outputWindowedValue capability to Java SDK
add ef0ee769e8f Merge pull request #29616: Add outputWindowedValue capability to Java SDK
add 19858e990ad [#29760] Only respond to sampling request while data sampling is enabled (#29761)
add a04a4ad4cd2 [yaml] Normalize drop on MapToFields
add c124e473fa4 Merge pull request #29755 from Polber/jkinard/mapping-drop
add a9f5ab14d06 Add support to use side inputs with Combine.PerKeyWithHotKeyFanout (#28867)
add def026bfe11 Bump google.golang.org/api from 0.153.0 to 0.154.0 in /sdks (#29734)
add 921e40a12f4 Dynamic SchemaTransform wrapper provider (#29561)
add 4264c2c3e25 Fixes breakages of the upgrade feature (#29731)
add 75746c6fe4a Bump cloud.google.com/go/storage from 1.35.1 to 1.36.0 in /sdks (#29767)
add 00462b13cc1 Bump google.golang.org/grpc from 1.59.0 to 1.60.0 in /sdks (#29714)
add 1ce707c3865 Add user-agent string for Beam in the GCS request header. (#29765)
add bc42a637d56 [#28543][prism] Implement State API (#29712)
add 9e8a310f0c0 Bump Precommit Java Examples Timeouts (#29771)
add c4c9bb82723 Add 2.54.0 release section to CHANGES.md (#29757)
add 543b013a548 Bump com.gradle.enterprise from 3.15.1 to 3.16 (#29740)
add a81c9e8e38b Bump actions/setup-python from 4 to 5 (#29654)
add 970a0bd9a92 Bump actions/stale from 8 to 9 (#29682)
add d297cd531ee Bump org.javacc.javacc from 3.0.0 to 3.0.2 (#29739)
add 1c7d178dd2b Bump github.com/google/uuid from 1.4.0 to 1.5.0 in /sdks (#29733)
add 52f4fc0c378 [Go SDK] Decrease sampling frequency for streaming jobs to avoid oversampling (#29774)
add 3bace64d16a Ignore testOnNewWorkerMetadata_correctlyRemovesStaleWindmillServers (#29744)
add 5fb4db31994 Bump actions/upload-artifact from 3 to 4 (#29777)
add 860bc30a110 Add SparkStructuredStreamingRunner to beam_PostCommit_Java_Tpcds_Spark workflow
add e1ce548cea7 Merge pull request #29781: Add SparkStructuredStreamingRunner to beam_PostCommit_Java_Tpcds_Spark workflow
add 539f04233c6 add alts to grpc vendored lib (#29763)
add 1033b71aff6 pin dind image temoporarily due to issues https://github.com/actions/actions-runner-controller/issues/3159 (#29796)
add fb8ae575eeb getProcessingTimesByStep returns a modifiable map (#29786)
add 7db8c274dd5 make DataSample external (#29788)
add f02bf88fd15 Upgrade download-artifact to v4 (#29801)
add c3d130c8ebf Update python base image dependencies (#29800)
add add34385719 Fix GenerateSequenceSchemaTransformProvider formatting (#29798)
add 4cfe5498850 Fix nil pointer access in logRuntimeDependencies (#29804)
add 2e38af69cf2 Adds Kafka upgrade module to the I/O precommit (#29773)
add 87794aaef03 Update dataflow container tag (#29790)
add 2baea8b65eb Bump golang.org/x/crypto from 0.14.0 to 0.17.0 in /.test-infra/mock-apis (#29809)
add d6426c4fdfc Bump google.golang.org/grpc from 1.60.0 to 1.60.1 in /sdks (#29814)
add 60f04081240 Update changes.md with known fixes. (#29817)
add 54d2c2df5cf Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sdks (#29810)
add 54a6b1f088e Publish job (#29685)
add 5ec9e7ec087 Fix lint
add a985c24ba84 Add computing approximate cost for dataflow jobs
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (db36ffbbadb)
\
N -- N -- N refs/heads/compute_cost (a985c24ba84)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 1 +
.github/actions/setup-action/action.yml | 6 +-
.../gh-actions-self-hosted-runners/arc/locals.tf | 2 +
.github/workflows/beam_Java_JMH.yml | 2 +
...eam_LoadTests_Java_CoGBK_Dataflow_Streaming.yml | 2 +-
.github/workflows/beam_Metrics_Report.yml | 2 +-
...PerformanceTests_BigQueryIO_Batch_Java_Avro.yml | 2 +-
...PerformanceTests_BigQueryIO_Batch_Java_Json.yml | 2 +-
..._PerformanceTests_BigQueryIO_Streaming_Java.yml | 2 +-
...m_PerformanceTests_SQLBigQueryIO_Batch_Java.yml | 2 +-
...PerformanceTests_WordCountIT_PythonVersions.yml | 2 +-
.github/workflows/beam_PostCommit_Java.yml | 2 +-
.../beam_PostCommit_Java_Avro_Versions.yml | 2 +-
.../beam_PostCommit_Java_BigQueryEarlyRollout.yml | 2 +-
.../workflows/beam_PostCommit_Java_DataflowV1.yml | 2 +-
.../workflows/beam_PostCommit_Java_DataflowV2.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 2 +-
...beam_PostCommit_Java_Examples_Dataflow_Java.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow_V2.yml | 2 +-
...m_PostCommit_Java_Examples_Dataflow_V2_Java.yml | 2 +-
.../beam_PostCommit_Java_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Java_Examples_Flink.yml | 2 +-
.../beam_PostCommit_Java_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Java_Hadoop_Versions.yml | 2 +-
.../beam_PostCommit_Java_IO_Performance_Tests.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Dataflow_Java11.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Dataflow_Java17.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Direct_Java11.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Direct_Java17.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Direct_Java21.yml | 4 +-
.../beam_PostCommit_Java_Jpms_Flink_Java11.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Spark_Java11.yml | 2 +-
.../beam_PostCommit_Java_Nexmark_Dataflow.yml | 2 +
.../beam_PostCommit_Java_Nexmark_Dataflow_V2.yml | 2 +
...am_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml | 2 +
.../beam_PostCommit_Java_Nexmark_Direct.yml | 2 +
.../beam_PostCommit_Java_Nexmark_Flink.yml | 2 +
.../beam_PostCommit_Java_Nexmark_Spark.yml | 2 +
.../beam_PostCommit_Java_PVR_Flink_Streaming.yml | 2 +-
.../workflows/beam_PostCommit_Java_PVR_Samza.yml | 2 +-
.../beam_PostCommit_Java_PVR_Spark3_Streaming.yml | 2 +-
.../beam_PostCommit_Java_PVR_Spark_Batch.yml | 4 +-
.github/workflows/beam_PostCommit_Java_Sickbay.yml | 2 +-
.../beam_PostCommit_Java_Tpcds_Dataflow.yml | 4 +-
.../workflows/beam_PostCommit_Java_Tpcds_Flink.yml | 2 +
.../workflows/beam_PostCommit_Java_Tpcds_Spark.yml | 12 +-
...am_PostCommit_Java_ValidatesRunner_Dataflow.yml | 2 +-
..._Java_ValidatesRunner_Dataflow_JavaVersions.yml | 2 +-
...mit_Java_ValidatesRunner_Dataflow_Streaming.yml | 2 +-
...PostCommit_Java_ValidatesRunner_Dataflow_V2.yml | 2 +-
..._Java_ValidatesRunner_Dataflow_V2_Streaming.yml | 2 +-
...beam_PostCommit_Java_ValidatesRunner_Direct.yml | 2 +-
...it_Java_ValidatesRunner_Direct_JavaVersions.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 2 +-
...ostCommit_Java_ValidatesRunner_Flink_Java11.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Samza.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Spark.yml | 2 +-
...va_ValidatesRunner_SparkStructuredStreaming.yml | 2 +-
...ostCommit_Java_ValidatesRunner_Spark_Java11.yml | 2 +-
...am_PostCommit_Java_ValidatesRunner_Twister2.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_ULR.yml | 2 +-
.github/workflows/beam_PostCommit_Javadoc.yml | 2 +-
.../beam_PostCommit_PortableJar_Flink.yml | 2 +-
.../beam_PostCommit_PortableJar_Spark.yml | 2 +-
.github/workflows/beam_PostCommit_Python.yml | 2 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 2 +-
.../beam_PostCommit_Python_Examples_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Python_Examples_Flink.yml | 2 +-
.../beam_PostCommit_Python_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Python_MongoDBIO_IT.yml | 2 +-
...stCommit_Python_ValidatesContainer_Dataflow.yml | 2 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 2 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Samza.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Spark.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Direct.yml | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 2 +-
.github/workflows/beam_PostCommit_SQL.yml | 2 +-
.../workflows/beam_PostCommit_Sickbay_Python.yml | 2 +-
.../beam_PostCommit_TransformService_Direct.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Direct.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Flink.yml | 2 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 2 +-
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 2 +-
..._PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml | 2 +-
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Samza.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Spark3.yml | 2 +-
.github/workflows/beam_PreCommit_ItFramework.yml | 2 +-
.github/workflows/beam_PreCommit_Java.yml | 4 +-
...eCommit_Java_Amazon-Web-Services2_IO_Direct.yml | 4 +-
...reCommit_Java_Amazon-Web-Services_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Amqp_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Azure_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Cassandra_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Cdap_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Clickhouse_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Csv_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Debezium_IO_Direct.yml | 4 +-
...beam_PreCommit_Java_ElasticSearch_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Examples_Dataflow.yml | 4 +-
...eam_PreCommit_Java_Examples_Dataflow_Java21.yml | 6 +-
...Commit_Java_File-schema-transform_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Flink_Versions.yml | 2 +-
.../beam_PreCommit_Java_GCP_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Google-ads_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_HBase_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_HCatalog_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Hadoop_IO_Direct.yml | 4 +-
.../workflows/beam_PreCommit_Java_IOs_Direct.yml | 4 +-
.../beam_PreCommit_Java_InfluxDb_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_JDBC_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Jms_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Kafka_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Kinesis_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Kudu_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_MongoDb_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Mqtt_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Neo4j_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_PVR_Flink_Batch.yml | 4 +-
.../beam_PreCommit_Java_PVR_Flink_Docker.yml | 2 +-
.../beam_PreCommit_Java_Parquet_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Pulsar_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_RabbitMq_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Redis_IO_Direct.yml | 4 +-
...am_PreCommit_Java_RequestResponse_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_SingleStore_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Snowflake_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Solr_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Spark3_Versions.yml | 2 +-
.../beam_PreCommit_Java_Splunk_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Thrift_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Tika_IO_Direct.yml | 4 +-
.github/workflows/beam_PreCommit_Python.yml | 2 +-
.../workflows/beam_PreCommit_Python_Coverage.yml | 2 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 2 +-
.../workflows/beam_PreCommit_Python_Examples.yml | 2 +-
.../beam_PreCommit_Python_Integration.yml | 2 +-
.../workflows/beam_PreCommit_Python_PVR_Flink.yml | 2 +-
.../workflows/beam_PreCommit_Python_Runners.yml | 2 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 2 +-
.github/workflows/beam_PreCommit_SQL.yml | 4 +-
.github/workflows/beam_PreCommit_SQL_Java11.yml | 4 +-
.github/workflows/beam_PreCommit_SQL_Java17.yml | 4 +-
.github/workflows/beam_PreCommit_Spotless.yml | 2 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 2 +-
.github/workflows/build_release_candidate.yml | 14 +-
.github/workflows/build_wheels.yml | 24 +-
.github/workflows/code_completion_plugin_tests.yml | 4 +-
.github/workflows/dask_runner_tests.yml | 8 +-
.github/workflows/java_tests.yml | 10 +-
.github/workflows/local_env_tests.yml | 4 +-
.github/workflows/python_dependency_tests.yml | 2 +-
.github/workflows/python_tests.yml | 14 +-
.github/workflows/run_perf_alert_tool.yml | 2 +-
.github/workflows/run_rc_validation.yml | 18 +-
.github/workflows/stale.yml | 2 +-
.github/workflows/typescript_tests.yml | 4 +-
.test-infra/mock-apis/go.mod | 6 +-
.test-infra/mock-apis/go.sum | 12 +-
CHANGES.md | 46 +-
build.gradle.kts | 1 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
.../apache/beam/gradle/GrpcVendoring_1_54_0.groovy | 1 +
.../org/apache/beam/examples/KafkaStreaming.java | 2 +-
examples/notebooks/beam-ml/README.md | 6 +-
.../beam-ml/automatic_model_refresh.ipynb | 10 -
examples/notebooks/beam-ml/mltransform_basic.ipynb | 2 +-
examples/notebooks/beam-ml/per_key_models.ipynb | 2 +-
.../notebooks/beam-ml/run_custom_inference.ipynb | 4 +-
.../run_inference_pytorch_tensorflow_sklearn.ipynb | 4 +-
.../beam-ml/speech_emotion_tensorflow.ipynb | 2 +-
gradle.properties | 4 +-
.../beam/it/jdbc/AbstractJDBCResourceManager.java | 6 +-
release/go-licenses/Dockerfile | 2 +-
release/src/main/Dockerfile | 2 +-
.../runners/core/construction/SplittableParDo.java | 10 +
.../construction/SplittableParDoNaiveBounded.java | 38 ++
.../core/construction/TransformUpgrader.java | 13 +-
...TimeBoundedSplittableProcessElementInvoker.java | 25 +-
.../apache/beam/runners/core/SimpleDoFnRunner.java | 79 ++-
.../core/metrics/ExecutionStateSampler.java | 10 +-
.../core/metrics/ExecutionStateTracker.java | 9 +-
runners/google-cloud-dataflow-java/build.gradle | 2 +-
.../beam/runners/dataflow/TestDataflowRunner.java | 14 +-
.../dataflow/worker/ActiveMessageMetadata.java | 32 +
.../dataflow/worker/DataflowExecutionContext.java | 61 +-
.../worker/DataflowExecutionStateSampler.java | 136 ++++
.../dataflow/worker/StreamingDataflowWorker.java | 25 +-
.../dataflow/worker/streaming/ActiveWorkState.java | 13 +-
.../worker/streaming/ComputationState.java | 10 +-
.../runners/dataflow/worker/streaming/Work.java | 74 ++-
.../worker/DataflowExecutionContextTest.java | 76 +++
.../worker/DataflowExecutionStateSamplerTest.java | 148 +++++
.../dataflow/worker/FakeWindmillServer.java | 10 +-
.../worker/StreamingDataflowWorkerTest.java | 89 ++-
.../worker/streaming/ActiveWorkStateTest.java | 4 +-
.../client/grpc/StreamingEngineClientTest.java | 2 +
.../worker/windmill/src/main/proto/windmill.proto | 16 +
sdks/go.mod | 27 +-
sdks/go.sum | 58 +-
sdks/go/container/Dockerfile | 2 +-
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/go/pkg/beam/core/runtime/exec/datasampler.go | 28 +-
.../pkg/beam/core/runtime/exec/datasampler_test.go | 24 +-
sdks/go/pkg/beam/core/runtime/exec/datasink.go | 4 +-
sdks/go/pkg/beam/core/runtime/exec/pcollection.go | 14 +-
sdks/go/pkg/beam/core/runtime/harness/harness.go | 22 +-
sdks/go/pkg/beam/io/natsio/common.go | 4 +
sdks/go/pkg/beam/io/natsio/end_estimator.go | 77 +++
sdks/go/pkg/beam/io/natsio/end_estimator_test.go | 78 +++
sdks/go/pkg/beam/io/natsio/example_test.go | 18 +
sdks/go/pkg/beam/io/natsio/helper_test.go | 48 +-
sdks/go/pkg/beam/io/natsio/read.go | 289 +++++++++
sdks/go/pkg/beam/io/natsio/read_option.go | 98 +++
sdks/go/pkg/beam/io/natsio/read_test.go | 212 +++++++
.../engine/data.go => io/natsio/time_policy.go} | 40 +-
.../io/natsio/{common.go => time_policy_test.go} | 55 +-
.../data.go => io/natsio/watermark_estimator.go} | 23 +-
.../pkg/beam/io/natsio/watermark_estimator_test.go | 67 ++
sdks/go/pkg/beam/io/natsio/write_test.go | 4 +-
.../pkg/beam/runners/prism/internal/engine/data.go | 176 +++++
.../prism/internal/engine/elementmanager.go | 226 ++++++-
.../runners/prism/internal/engine/engine_test.go | 159 +++++
sdks/go/pkg/beam/runners/prism/internal/execute.go | 22 +
.../beam/runners/prism/internal/execute_test.go | 8 +-
.../pkg/beam/runners/prism/internal/handlepardo.go | 9 +-
.../beam/runners/prism/internal/jobservices/job.go | 3 +-
.../prism/internal/jobservices/management.go | 20 +-
.../pkg/beam/runners/prism/internal/preprocess.go | 14 +-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 22 +-
.../runners/prism/internal/unimplemented_test.go | 48 +-
.../pkg/beam/runners/prism/internal/urns/urns.go | 5 +
.../beam/runners/prism/internal/worker/bundle.go | 6 +-
.../beam/runners/prism/internal/worker/worker.go | 89 ++-
sdks/go/test/integration/primitives/state.go | 53 +-
.../java/org/apache/beam/sdk/io/FileSystems.java | 17 +
.../GenerateSequenceSchemaTransformProvider.java | 201 ++++++
.../apache/beam/sdk/providers/package-info.java | 23 +
.../org/apache/beam/sdk/transforms/Combine.java | 24 +-
.../org/apache/beam/sdk/transforms/Create.java | 191 ++++++
.../java/org/apache/beam/sdk/transforms/DoFn.java | 58 ++
.../beam/sdk/transforms/DoFnOutputReceivers.java | 27 +
.../org/apache/beam/sdk/transforms/DoFnTester.java | 22 +
.../sdk/transforms/windowing/GlobalWindow.java | 5 +
.../org/apache/beam/sdk/util/HistogramData.java | 12 +
.../org/apache/beam/sdk/io/FileSystemsTest.java | 20 +
...enerateSequenceSchemaTransformProviderTest.java | 61 ++
.../apache/beam/sdk/transforms/CombineTest.java | 24 +
.../org/apache/beam/sdk/transforms/CreateTest.java | 81 +++
.../sdk/transforms/reflect/DoFnInvokersTest.java | 22 +
.../apache/beam/sdk/util/HistogramDataTest.java | 26 +
.../beam/sdk/extensions/gcp/util/GcsUtilIT.java | 4 +-
.../extensions/sql/zetasql/BeamZetaSqlCalcRel.java | 12 +
.../org/apache/beam/sdk/fn/JvmInitializers.java | 2 +
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 295 ++++++++-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 2 +-
.../sdk/io/gcp/bigquery/BigQueryIOTranslation.java | 705 +++++++++++----------
.../sdk/io/gcp/bigquery/BigQuerySinkMetrics.java | 61 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 21 +
.../sdk/io/gcp/bigtable/BigtableServiceImpl.java | 9 +-
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 1 +
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 12 +
.../sdk/io/gcp/bigquery/BigQueryClusteringIT.java | 4 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIOJsonIT.java | 4 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIOReadIT.java | 4 +-
.../bigquery/BigQueryIOStorageReadTableRowIT.java | 4 +-
.../beam/sdk/io/gcp/bigquery/BigQueryKmsKeyIT.java | 4 +-
.../io/gcp/bigquery/BigQueryNestedRecordsIT.java | 4 +-
.../bigquery/BigQuerySchemaUpdateOptionsIT.java | 7 +-
.../io/gcp/bigquery/BigQuerySinkMetricsTest.java | 44 ++
.../BigQueryTimePartitioningClusteringIT.java | 4 +-
.../sdk/io/gcp/bigquery/BigQueryToTableIT.java | 10 +-
.../io/gcp/bigtable/BigtableServiceImplTest.java | 63 ++
.../beam/sdk/io/gcp/storage/GcsKmsKeyIT.java | 3 +-
.../apache/beam/sdk/io/gcp/storage/GcsMatchIT.java | 3 +-
.../beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java | 11 +
sdks/java/io/kafka/upgrade/build.gradle | 2 -
.../sdk/io/kafka/upgrade/KafkaIOTranslation.java | 414 ++++++------
.../io/kafka/upgrade/KafkaIOTranslationTest.java | 11 +
.../beam/sdk/io/pulsar/ReadFromPulsarDoFnTest.java | 13 +
.../ReadFromSparkReceiverWithOffsetDoFnTest.java | 13 +
.../apache_beam/examples/inference/README.md | 2 +-
.../io/external/xlang_kafkaio_it_test.py | 4 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 7 +-
sdks/python/apache_beam/ml/inference/__init__.py | 2 +-
sdks/python/apache_beam/ml/inference/base.py | 14 +-
.../ml/inference/huggingface_inference.py | 19 +-
sdks/python/apache_beam/ml/transforms/tft.py | 28 +-
sdks/python/apache_beam/ml/transforms/tft_test.py | 81 ++-
.../runners/dataflow/internal/apiclient.py | 6 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../apache_beam/runners/interactive/utils.py | 8 +-
.../runners/portability/sdk_container_builder.py | 6 +-
.../apache_beam/runners/worker/sdk_worker.py | 22 +-
.../testing/load_tests/load_test_metrics_utils.py | 2 +
sdks/python/apache_beam/transforms/external.py | 1 +
.../external_schematransform_provider.py | 277 ++++++++
.../external_schematransform_provider_test.py | 140 ++++
sdks/python/apache_beam/typehints/schemas.py | 4 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/yaml_mapping.py | 13 +-
sdks/python/apache_beam/yaml/yaml_transform.py | 2 +
sdks/python/container/Dockerfile | 2 +-
sdks/python/container/boot.go | 9 +-
.../container/py310/base_image_requirements.txt | 68 +-
.../container/py311/base_image_requirements.txt | 66 +-
.../container/py38/base_image_requirements.txt | 66 +-
.../container/py39/base_image_requirements.txt | 68 +-
sdks/python/expansion-service-container/Dockerfile | 2 +-
sdks/python/pytest.ini | 2 +-
sdks/python/setup.py | 2 +-
sdks/python/test-suites/dataflow/build.gradle | 4 +-
sdks/python/test-suites/direct/build.gradle | 8 +-
sdks/python/test-suites/direct/common.gradle | 2 +-
sdks/python/test-suites/gradle.properties | 4 +-
sdks/python/test-suites/xlang/build.gradle | 9 +-
sdks/typescript/package.json | 2 +-
settings.gradle.kts | 6 +-
.../site/content/en/documentation/ml/about-ml.md | 2 +-
.../en/documentation/ml/inference-overview.md | 4 +-
.../en/documentation/ml/multi-model-pipelines.md | 2 +-
.../site/content/en/documentation/ml/overview.md | 4 +-
.../en/documentation/runtime/environments.md | 10 +-
.../site/content/en/documentation/sdks/python.md | 2 +-
.../transforms/python/elementwise/runinference.md | 2 +-
.../en/get-started/resources/learning-resources.md | 2 +-
331 files changed, 5783 insertions(+), 1346 deletions(-)
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/ActiveMessageMetadata.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateSampler.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/DataflowExecutionStateSamplerTest.java
create mode 100644 sdks/go/pkg/beam/io/natsio/end_estimator.go
create mode 100644 sdks/go/pkg/beam/io/natsio/end_estimator_test.go
create mode 100644 sdks/go/pkg/beam/io/natsio/read.go
create mode 100644 sdks/go/pkg/beam/io/natsio/read_option.go
create mode 100644 sdks/go/pkg/beam/io/natsio/read_test.go
copy sdks/go/pkg/beam/{runners/prism/internal/engine/data.go => io/natsio/time_policy.go} (57%)
copy sdks/go/pkg/beam/io/natsio/{common.go => time_policy_test.go} (52%)
copy sdks/go/pkg/beam/{runners/prism/internal/engine/data.go => io/natsio/watermark_estimator.go} (67%)
create mode 100644 sdks/go/pkg/beam/io/natsio/watermark_estimator_test.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/engine/engine_test.go
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/providers/GenerateSequenceSchemaTransformProvider.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/providers/package-info.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/providers/GenerateSequenceSchemaTransformProviderTest.java
create mode 100644 sdks/python/apache_beam/transforms/external_schematransform_provider.py
create mode 100644 sdks/python/apache_beam/transforms/external_schematransform_provider_test.py