You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2023/08/11 05:08:26 UTC
[beam] branch nightly-refs/heads/master updated (6f60a6ce5ed -> 761aa7f06b5)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 6f60a6ce5ed Fix stateful processing using direct runner with type checks enabled (#27646)
add 71e060c1ebd Update how we specify artifact modes (#27939)
add b5f260dfed7 add expose gcloud kubelet path to setup action (#27952)
add e3c58d45cf5 Pre 2.50.0 Cut image requirement update. (#27947)
add f392a2153ee Bump github.com/fsouza/fake-gcs-server from 1.47.3 to 1.47.4 in /sdks (#27871)
add ba04f6098f7 Merge pull request #27866: Allow writing protos directly to the storage API without conversion
add fe974de5aee Fix setup action checkout (#27954)
add b34abc44ec3 Refactor terraform GKE and add bitnami kafka module (#27949)
add c994fddd05a Update BOM (#27956)
add 772628388d8 Also use config pattern for yaml providers.
add 6b2d395de76 Refactor yaml providers for more eager and explicit errors.
add 051872e6deb Add mypy typing to registration dict.
add 47e8de3aed0 Merge pull request #27884 Also use config pattern for yaml providers.
add 38b281e37a7 Moving to 2.51.0-SNAPSHOT on master branch.
add 9e311ad3aed Add missing notebooks to Readme (#27960)
add 3c5c7283ba4 Adds documentation regarding the Transform Service (#27946)
add 761aa7f06b5 Initial pass at modernizing the Runner author guide. (#27632)
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 1 +
.github/actions/setup-action/action.yml | 25 +-
.github/workflows/beam_PreCommit_RAT.yml | 2 +-
.test-infra/kafka/bitnami/README.md | 113 +++++
.test-infra/kafka/bitnami/kafka.tf | 114 +++++
.../02.redis => kafka/bitnami}/provider.tf | 2 +-
.../google-kubernetes-engine/README.md | 44 +-
.../apache-beam-testing.tfvars | 5 +-
.../{modules/03-cluster => }/cluster.tf | 27 +-
.../google-kubernetes-engine/main.tf | 73 ---
.../modules/01-setup/README.md | 26 -
.../modules/01-setup/iam.tf | 39 --
.../modules/01-setup/output.tf | 22 -
.../modules/01-setup/variables.tf | 32 --
.../modules/02-network/README.md | 26 -
.../modules/02-network/nat.tf | 43 --
.../modules/02-network/network.tf | 65 ---
.../modules/02-network/output.tf | 37 --
.../modules/02-network/provider.tf | 22 -
.../modules/02-network/services.tf | 23 -
.../modules/02-network/variables.tf | 44 --
.../modules/03-cluster/README.md | 22 -
.../modules/03-cluster/provider.tf | 22 -
.../modules/03-cluster/services.tf | 23 -
.../modules/03-cluster/variables.tf | 53 ---
.../modules/04-bastion/README.md | 22 -
.../modules/04-bastion/compute.tf | 64 ---
.../modules/04-bastion/data.tf | 44 --
.../modules/04-bastion/provider.tf | 22 -
.../modules/04-bastion/services.tf | 23 -
.../modules/04-bastion/tinyproxy.conf | 368 --------------
.../modules/04-bastion/variables.tf | 67 ---
.../01-setup/services.tf => prerequisites.tf} | 27 +-
.../{modules/01-setup => }/provider.tf | 0
.../google-kubernetes-engine/variables.tf | 17 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 10 +-
examples/notebooks/beam-ml/README.md | 6 +
gradle.properties | 4 +-
release/src/main/scripts/jenkins_jobs.txt | 2 +-
sdks/go.mod | 2 +-
sdks/go.sum | 4 +-
sdks/go/pkg/beam/core/core.go | 2 +-
.../container/license_scripts/dep_urls_java.yaml | 2 +-
.../beam/sdk/io/gcp/bigquery/AppendClientInfo.java | 32 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 96 ++++
.../beam/sdk/io/gcp/bigquery/BigQueryServices.java | 5 +-
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 8 +-
.../bigquery/StorageApiDynamicDestinations.java | 3 +
.../StorageApiDynamicDestinationsBeamRow.java | 6 +
...StorageApiDynamicDestinationsGenericRecord.java | 7 +
.../StorageApiDynamicDestinationsProto.java | 100 ++++
.../StorageApiDynamicDestinationsTableRow.java | 7 +
.../bigquery/StorageApiWriteUnshardedRecords.java | 66 ++-
.../bigquery/StorageApiWritesShardedRecords.java | 29 +-
.../io/gcp/bigquery/TableRowToStorageApiProto.java | 83 +++-
.../sdk/io/gcp/testing/FakeDatasetService.java | 7 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 111 ++++-
.../bigquery/StorageApiDirectWriteProtosIT.java | 197 ++++++++
.../bigquery/TableRowToStorageApiProtoTest.java | 418 +++++++++++++++-
.../examples/ml_transform/ml_transform_basic.py | 28 +-
sdks/python/apache_beam/ml/transforms/base.py | 47 +-
sdks/python/apache_beam/ml/transforms/base_test.py | 19 +-
sdks/python/apache_beam/ml/transforms/handlers.py | 9 +-
sdks/python/apache_beam/ml/transforms/tft_test.py | 37 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/pipeline.schema.yaml | 1 +
.../apache_beam/yaml/standard_providers.yaml | 5 +-
sdks/python/apache_beam/yaml/yaml_provider.py | 122 +++--
.../container/py310/base_image_requirements.txt | 22 +-
.../container/py311/base_image_requirements.txt | 20 +-
.../container/py38/base_image_requirements.txt | 22 +-
.../container/py39/base_image_requirements.txt | 22 +-
sdks/typescript/package.json | 2 +-
.../www/site/content/en/contribute/runner-guide.md | 529 ++++++++-------------
.../content/en/documentation/programming-guide.md | 70 +++
.../www/site/static/images/transform_service.png | Bin 0 -> 41817 bytes
76 files changed, 1939 insertions(+), 1784 deletions(-)
create mode 100644 .test-infra/kafka/bitnami/README.md
create mode 100644 .test-infra/kafka/bitnami/kafka.tf
copy .test-infra/{pipelines/infrastructure/03.io/api-overuse-study/02.redis => kafka/bitnami}/provider.tf (99%)
copy .test-infra/{pipelines/infrastructure/01.setup => terraform/google-cloud-platform/google-kubernetes-engine}/apache-beam-testing.tfvars (88%)
rename .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/{modules/03-cluster => }/cluster.tf (64%)
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/main.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/01-setup/README.md
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/01-setup/iam.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/01-setup/output.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/01-setup/variables.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/02-network/README.md
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/02-network/nat.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/02-network/network.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/02-network/output.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/02-network/provider.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/02-network/services.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/02-network/variables.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/03-cluster/README.md
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/03-cluster/provider.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/03-cluster/services.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/03-cluster/variables.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/04-bastion/README.md
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/04-bastion/compute.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/04-bastion/data.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/04-bastion/provider.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/04-bastion/services.tf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/04-bastion/tinyproxy.conf
delete mode 100644 .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/modules/04-bastion/variables.tf
rename .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/{modules/01-setup/services.tf => prerequisites.tf} (60%)
rename .test-infra/terraform/google-cloud-platform/google-kubernetes-engine/{modules/01-setup => }/provider.tf (100%)
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiDynamicDestinationsProto.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiDirectWriteProtosIT.java
create mode 100644 website/www/site/static/images/transform_service.png