You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2023/12/20 03:45:56 UTC
(beam) branch nightly-refs/heads/master updated (52f4fc0c378 -> 0bbf2c3351b)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 52f4fc0c378 [Go SDK] Decrease sampling frequency for streaming jobs to avoid oversampling (#29774)
add 3bace64d16a Ignore testOnNewWorkerMetadata_correctlyRemovesStaleWindmillServers (#29744)
add 5fb4db31994 Bump actions/upload-artifact from 3 to 4 (#29777)
add 860bc30a110 Add SparkStructuredStreamingRunner to beam_PostCommit_Java_Tpcds_Spark workflow
add e1ce548cea7 Merge pull request #29781: Add SparkStructuredStreamingRunner to beam_PostCommit_Java_Tpcds_Spark workflow
add 539f04233c6 add alts to grpc vendored lib (#29763)
add 1033b71aff6 pin dind image temoporarily due to issues https://github.com/actions/actions-runner-controller/issues/3159 (#29796)
add fb8ae575eeb getProcessingTimesByStep returns a modifiable map (#29786)
add 7db8c274dd5 make DataSample external (#29788)
add f02bf88fd15 Upgrade download-artifact to v4 (#29801)
add c3d130c8ebf Update python base image dependencies (#29800)
add add34385719 Fix GenerateSequenceSchemaTransformProvider formatting (#29798)
add 4cfe5498850 Fix nil pointer access in logRuntimeDependencies (#29804)
add 2e38af69cf2 Adds Kafka upgrade module to the I/O precommit (#29773)
add 87794aaef03 Update dataflow container tag (#29790)
add 2baea8b65eb Bump golang.org/x/crypto from 0.14.0 to 0.17.0 in /.test-infra/mock-apis (#29809)
add d6426c4fdfc Bump google.golang.org/grpc from 1.60.0 to 1.60.1 in /sdks (#29814)
add 60f04081240 Update changes.md with known fixes. (#29817)
add 54d2c2df5cf Bump golang.org/x/crypto from 0.16.0 to 0.17.0 in /sdks (#29810)
add f841098c54d Support DatabaseID in Datastore beam connector (#29815)
add 1a7174c68bd Build and publish multi-arch wheels separately from main wheels (#29821)
add b9a62f0f793 [YAML] Properly deligate methods for renaming provider.
add 56d177713ed Merge pull request #29827 [YAML] Properly deligate methods for renaming provider.
add c5d53d61e78 Revert to non-schema-based Sql transform.
add 0bbf2c3351b Merge pull request #29826 Revert to non-schema-based Sql transform.
No new revisions were added by this update.
Summary of changes:
.../gh-actions-self-hosted-runners/arc/locals.tf | 2 +
...eam_LoadTests_Java_CoGBK_Dataflow_Streaming.yml | 2 +-
.github/workflows/beam_Metrics_Report.yml | 2 +-
...PerformanceTests_BigQueryIO_Batch_Java_Avro.yml | 2 +-
...PerformanceTests_BigQueryIO_Batch_Java_Json.yml | 2 +-
..._PerformanceTests_BigQueryIO_Streaming_Java.yml | 2 +-
...m_PerformanceTests_SQLBigQueryIO_Batch_Java.yml | 2 +-
...PerformanceTests_WordCountIT_PythonVersions.yml | 2 +-
.github/workflows/beam_PostCommit_Java.yml | 2 +-
.../beam_PostCommit_Java_Avro_Versions.yml | 2 +-
.../beam_PostCommit_Java_BigQueryEarlyRollout.yml | 2 +-
.../workflows/beam_PostCommit_Java_DataflowV1.yml | 2 +-
.../workflows/beam_PostCommit_Java_DataflowV2.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 2 +-
...beam_PostCommit_Java_Examples_Dataflow_Java.yml | 2 +-
.../beam_PostCommit_Java_Examples_Dataflow_V2.yml | 2 +-
...m_PostCommit_Java_Examples_Dataflow_V2_Java.yml | 2 +-
.../beam_PostCommit_Java_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Java_Examples_Flink.yml | 2 +-
.../beam_PostCommit_Java_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Java_Hadoop_Versions.yml | 2 +-
.../beam_PostCommit_Java_IO_Performance_Tests.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Dataflow_Java11.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Dataflow_Java17.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Direct_Java11.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Direct_Java17.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Direct_Java21.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Flink_Java11.yml | 2 +-
.../beam_PostCommit_Java_Jpms_Spark_Java11.yml | 2 +-
.../beam_PostCommit_Java_PVR_Flink_Streaming.yml | 2 +-
.../workflows/beam_PostCommit_Java_PVR_Samza.yml | 2 +-
.../beam_PostCommit_Java_PVR_Spark3_Streaming.yml | 2 +-
.../beam_PostCommit_Java_PVR_Spark_Batch.yml | 4 +-
.github/workflows/beam_PostCommit_Java_Sickbay.yml | 2 +-
.../workflows/beam_PostCommit_Java_Tpcds_Spark.yml | 10 +-
...am_PostCommit_Java_ValidatesRunner_Dataflow.yml | 2 +-
..._Java_ValidatesRunner_Dataflow_JavaVersions.yml | 2 +-
...mit_Java_ValidatesRunner_Dataflow_Streaming.yml | 2 +-
...PostCommit_Java_ValidatesRunner_Dataflow_V2.yml | 2 +-
..._Java_ValidatesRunner_Dataflow_V2_Streaming.yml | 2 +-
...beam_PostCommit_Java_ValidatesRunner_Direct.yml | 2 +-
...it_Java_ValidatesRunner_Direct_JavaVersions.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 2 +-
...ostCommit_Java_ValidatesRunner_Flink_Java11.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Samza.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Spark.yml | 2 +-
...va_ValidatesRunner_SparkStructuredStreaming.yml | 2 +-
...ostCommit_Java_ValidatesRunner_Spark_Java11.yml | 2 +-
...am_PostCommit_Java_ValidatesRunner_Twister2.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_ULR.yml | 2 +-
.github/workflows/beam_PostCommit_Javadoc.yml | 2 +-
.../beam_PostCommit_PortableJar_Flink.yml | 2 +-
.../beam_PostCommit_PortableJar_Spark.yml | 2 +-
.github/workflows/beam_PostCommit_Python.yml | 2 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 2 +-
.../beam_PostCommit_Python_Examples_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Python_Examples_Flink.yml | 2 +-
.../beam_PostCommit_Python_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Python_MongoDBIO_IT.yml | 2 +-
...stCommit_Python_ValidatesContainer_Dataflow.yml | 2 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 2 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Samza.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Spark.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Direct.yml | 2 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 2 +-
.github/workflows/beam_PostCommit_SQL.yml | 2 +-
.../workflows/beam_PostCommit_Sickbay_Python.yml | 2 +-
.../beam_PostCommit_TransformService_Direct.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Direct.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Flink.yml | 2 +-
.../beam_PostCommit_XVR_GoUsingJava_Dataflow.yml | 2 +-
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 2 +-
..._PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml | 2 +-
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Samza.yml | 2 +-
.github/workflows/beam_PostCommit_XVR_Spark3.yml | 2 +-
.github/workflows/beam_PreCommit_ItFramework.yml | 2 +-
.github/workflows/beam_PreCommit_Java.yml | 4 +-
...eCommit_Java_Amazon-Web-Services2_IO_Direct.yml | 4 +-
...reCommit_Java_Amazon-Web-Services_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Amqp_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Azure_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Cassandra_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Cdap_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Clickhouse_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Csv_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Debezium_IO_Direct.yml | 4 +-
...beam_PreCommit_Java_ElasticSearch_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Examples_Dataflow.yml | 2 +-
...eam_PreCommit_Java_Examples_Dataflow_Java21.yml | 4 +-
...Commit_Java_File-schema-transform_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Flink_Versions.yml | 2 +-
.../beam_PreCommit_Java_GCP_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Google-ads_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_HBase_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_HCatalog_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Hadoop_IO_Direct.yml | 4 +-
.../workflows/beam_PreCommit_Java_IOs_Direct.yml | 4 +-
.../beam_PreCommit_Java_InfluxDb_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_JDBC_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Jms_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Kafka_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Kinesis_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Kudu_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_MongoDb_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Mqtt_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Neo4j_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_PVR_Flink_Batch.yml | 4 +-
.../beam_PreCommit_Java_PVR_Flink_Docker.yml | 2 +-
.../beam_PreCommit_Java_Parquet_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Pulsar_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_RabbitMq_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Redis_IO_Direct.yml | 4 +-
...am_PreCommit_Java_RequestResponse_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_SingleStore_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Snowflake_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Solr_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Spark3_Versions.yml | 2 +-
.../beam_PreCommit_Java_Splunk_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Thrift_IO_Direct.yml | 4 +-
.../beam_PreCommit_Java_Tika_IO_Direct.yml | 4 +-
.github/workflows/beam_PreCommit_Python.yml | 2 +-
.../workflows/beam_PreCommit_Python_Coverage.yml | 2 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 2 +-
.../workflows/beam_PreCommit_Python_Examples.yml | 2 +-
.../beam_PreCommit_Python_Integration.yml | 2 +-
.../workflows/beam_PreCommit_Python_PVR_Flink.yml | 2 +-
.../workflows/beam_PreCommit_Python_Runners.yml | 2 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 2 +-
.github/workflows/beam_PreCommit_SQL.yml | 4 +-
.github/workflows/beam_PreCommit_SQL_Java11.yml | 4 +-
.github/workflows/beam_PreCommit_SQL_Java17.yml | 4 +-
.github/workflows/beam_PreCommit_Spotless.yml | 2 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 2 +-
.github/workflows/build_wheels.yml | 30 +-
.github/workflows/code_completion_plugin_tests.yml | 2 +-
.github/workflows/dask_runner_tests.yml | 4 +-
.github/workflows/java_tests.yml | 10 +-
.github/workflows/python_tests.yml | 6 +-
.test-infra/mock-apis/go.mod | 6 +-
.test-infra/mock-apis/go.sum | 12 +-
CHANGES.md | 11 +-
build.gradle.kts | 1 +
.../apache/beam/gradle/GrpcVendoring_1_54_0.groovy | 1 +
.../dataflow/worker/DataflowExecutionContext.java | 5 +-
.../worker/DataflowExecutionStateSampler.java | 5 +-
.../worker/DataflowExecutionContextTest.java | 2 +-
.../worker/DataflowExecutionStateSamplerTest.java | 10 +-
.../client/grpc/StreamingEngineClientTest.java | 2 +
sdks/go.mod | 4 +-
sdks/go.sum | 8 +-
sdks/go/pkg/beam/core/runtime/exec/datasampler.go | 28 +-
.../pkg/beam/core/runtime/exec/datasampler_test.go | 24 +-
.../GenerateSequenceSchemaTransformProvider.java | 4 +-
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 349 ++++++++++++++++++---
.../beam/sdk/io/gcp/datastore/DatastoreV1Test.java | 94 +++++-
.../beam/sdk/io/gcp/datastore/SplitQueryFnIT.java | 17 +-
.../apache/beam/sdk/io/gcp/datastore/V1ReadIT.java | 26 +-
.../beam/sdk/io/gcp/datastore/V1TestUtil.java | 59 +++-
.../beam/sdk/io/gcp/datastore/V1WriteIT.java | 11 +-
.../apache_beam/runners/dataflow/internal/names.py | 2 +-
.../apache_beam/yaml/standard_providers.yaml | 8 +-
sdks/python/apache_beam/yaml/yaml_provider.py | 5 +-
sdks/python/container/boot.go | 9 +-
.../container/py310/base_image_requirements.txt | 66 ++--
.../container/py311/base_image_requirements.txt | 64 ++--
.../container/py38/base_image_requirements.txt | 64 ++--
.../container/py39/base_image_requirements.txt | 66 ++--
173 files changed, 883 insertions(+), 506 deletions(-)