You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2023/02/17 04:35:22 UTC
[beam] branch nightly-refs/heads/master updated (d8fbbf03a79 -> 40838f76447)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from d8fbbf03a79 Update python container images (#25475)
add fce8b6555ce Fix beam.Row.__eq__ for rows with trailing columns (#23876)
add 3edf78996cd TFMA notebook showing ExtractEvaluateAndWriteResult Tranfsorm (#25381)
add 9e737d38f84 Fix whitespace check (#25514)
add 6e08092bd63 Add back links removed in #24744 (#25513)
add 2aa77521413 Fix pull_licenses_java returning 404 from opensource.org
add 1b1cc449ef8 Merge pull request #25515: Fix pull_licenses_java returning 404 from opensource.org
add 9fb9d2cb415 [#24789][prism] internal/jobservices metrics + tests (#25497)
add 29ea6e0eb8a [Go SDK]: Allow SDF methods to have context param and error return value (#25437)
add 5fc19fffcdf Move closing milestone out of PMC-only tasks (#25516)
add a82084ce87b update GCP cloud libraries BOM to 26.8.0 (#25470)
add c210df3d3b3 Fix interface{} in iter& emit type of DoFn in Go (#25203)
add 545a2e63ca7 Task #25064: Python SDK Data sampling implementation (#25093)
add 4e6d751f92e Add support for all Java based portable runners to consume elements embedding on the control response. (#25463)
add 684917f9bc5 Swap Java SDK container to use eclipse-temurin as the base instead of OpenJDK (#25402)
add d391bf2b3a5 Update BigQueryIO documentation with details on how to override the query execution project (#25400)
add 40838f76447 Support Avro GenericRecord as a valid format for StorageWrite API on BigQueryIO (#24274)
No new revisions were added by this update.
Summary of changes:
CHANGES.md | 4 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 10 +-
examples/notebooks/beam-ml/tfma_beam.ipynb | 742 +++++++++++++++++++++
.../control/DefaultJobBundleFactory.java | 3 +
.../fnexecution/control/SdkHarnessClient.java | 4 +
.../environment/EmbeddedEnvironmentFactory.java | 7 +-
sdks/go/pkg/beam/core/funcx/output.go | 5 +-
sdks/go/pkg/beam/core/funcx/output_test.go | 2 +
sdks/go/pkg/beam/core/funcx/sideinput.go | 3 +
sdks/go/pkg/beam/core/funcx/sideinput_test.go | 1 +
sdks/go/pkg/beam/core/graph/fn.go | 108 ++-
sdks/go/pkg/beam/core/graph/fn_test.go | 87 +++
sdks/go/pkg/beam/core/runtime/exec/datasource.go | 8 +-
.../pkg/beam/core/runtime/exec/datasource_test.go | 12 +-
sdks/go/pkg/beam/core/runtime/exec/sdf.go | 134 ++--
sdks/go/pkg/beam/core/runtime/exec/sdf_invokers.go | 339 ++++------
.../beam/core/runtime/exec/sdf_invokers_arity.go | 336 ++++++++++
.../beam/core/runtime/exec/sdf_invokers_arity.tmpl | 246 +++++++
.../beam/core/runtime/exec/sdf_invokers_test.go | 250 ++++++-
sdks/go/pkg/beam/core/runtime/exec/sdf_test.go | 17 +-
sdks/go/pkg/beam/io/parquetio/parquetio.go | 4 +-
.../beam/runners/prism/internal/config/config.go | 2 +-
.../runners/prism/internal/jobservices/metrics.go | 497 ++++++++++++++
.../prism/internal/jobservices/metrics_test.go | 135 ++++
.../beam/runners/prism/internal/urns/urns_test.go | 2 +-
sdks/java/container/Dockerfile | 2 +-
.../container/license_scripts/dep_urls_java.yaml | 2 +-
.../license_scripts/pull_licenses_java.py | 4 +
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 20 +-
.../extensions/avro/schemas/utils/AvroUtils.java | 14 +-
.../fn/harness/jmh/ProcessBundleBenchmark.java | 16 +-
.../fn/harness/jmh/ProcessBundleBenchmarkTest.java | 18 +-
.../AvroGenericRecordToStorageApiProto.java | 378 +++++++++++
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 50 +-
.../beam/sdk/io/gcp/bigquery/RowWriterFactory.java | 4 +
...torageApiDynamicDestinationsGenericRecord.java} | 67 +-
.../AvroGenericRecordToStorageApiProtoTest.java | 473 +++++++++++++
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 109 ++-
sdks/python/apache_beam/pvalue.py | 7 +-
sdks/python/apache_beam/pvalue_test.py | 18 +
.../apache_beam/runners/worker/bundle_processor.py | 95 ++-
.../runners/worker/bundle_processor_test.py | 132 ++++
.../apache_beam/runners/worker/data_sampler.py | 156 +++++
.../runners/worker/data_sampler_test.py | 192 ++++++
.../apache_beam/runners/worker/sdk_worker.py | 34 +-
.../apache_beam/runners/worker/sdk_worker_main.py | 9 +-
.../apache_beam/runners/worker/sdk_worker_test.py | 55 +-
sdks/python/apache_beam/transforms/environments.py | 1 +
.../site/content/en/contribute/release-guide.md | 8 +-
.../documentation/io/built-in/google-bigquery.md | 5 +
.../site/content/en/documentation/io/connectors.md | 4 +-
.../www/site/content/en/get-started/downloads.md | 2 +-
52 files changed, 4406 insertions(+), 427 deletions(-)
create mode 100755 examples/notebooks/beam-ml/tfma_beam.ipynb
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/sdf_invokers_arity.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/sdf_invokers_arity.tmpl
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/jobservices/metrics.go
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/jobservices/metrics_test.go
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroGenericRecordToStorageApiProto.java
copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/{StorageApiDynamicDestinationsBeamRow.java => StorageApiDynamicDestinationsGenericRecord.java} (50%)
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/AvroGenericRecordToStorageApiProtoTest.java
create mode 100644 sdks/python/apache_beam/runners/worker/data_sampler.py
create mode 100644 sdks/python/apache_beam/runners/worker/data_sampler_test.py