You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2022/12/01 04:38:43 UTC
[beam] branch nightly-refs/heads/master updated (fd7e86269fa -> 8349ee6c12a)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from fd7e86269fa Update java-multi-language-pipelines.md (#24345)
add 0fe001ee9eb [Tour Of Beam] persistence_key for Pg::SaveSnippet (#24287)
add 6dc4b73ff0b Get postcommits green and unsickbay (#24342)
add 11151544d80 Fix workflow cron syntax (#24376)
add d31362fd47d concurrency (#24332)
add 0a3fff05139 Check for null in BeamFnDataGrpcMultiplexer
add a6c645daa5f Upgrade checkerframework to 3.12.0
add a493cc12b46 Merge pull request #24247: Upgrade checkerframework to 3.12.0
add 3fee48db29a Upgrade checkerframework gradle plugin to 0.6.19
add e9f73c47d01 Merge pull request #24239: Upgrade checkerframework gradle plugin to 0.6.19
add f68db1fcab6 Handle CompleteWorkStatus shutdown signal
add 7ee09bca3d5 Merge pull request #24188: Handle CompleteWorkStatus shutdown signal
add 3477d4a76d4 Revert "Force discarding mode in with_fanout without rewindowing." (#24378)
add c3bb33aafbc Bump pillow from 9.2.0 to 9.3.0 in /sdks/python/apache_beam/examples/ml-orchestration/kfp/components/preprocessing (#24173)
add 135007ec6b4 Update precombine bencmark to better represent varied workloads (#24343)
add ec6d163cf36 Merge pull request #24320: update bom to the latest one
add 5bb13fa35b9 Merge pull request #24147: First step in adding schema update to Storage API sink. Refactor code #21395
add 928400c4a30 Simplify sdks/java/harness build
add cb1dcf93a6c Move configuration changes before shadowJar
add 157e9fe7989 Exclude :sdks:java:core from harness jar
add d98d2088c1a Enable shadowJar validation for sdks/java/harness
add 163fd65346e Add missing portability runner dependencies
add 127397f4e6d Exclude jamm from harness jar
add 4720f0073db Enforce GCP BOM on sdks/java/harness
add 37fb90c98d1 Merge pull request #24264: Simplify sdks/java/harness shadowJar
add 4b959e73a71 Fix SparkReceiverIOIT test (#24375)
add f4a18c53f26 Bump cloud.google.com/go/bigquery from 1.42.0 to 1.43.0 in /sdks (#24311)
add b9088ba62a2 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#24348)
add 60719d1b4d7 [Spark Dataset runner] Fix support for Java 11 (closes #24392)
add 36b89946883 fix spotless
add db54c84e170 Merge pull request #24393: [Spark Dataset runner] Fix support for Java 11
add 22dcacb163b pg_23079 remove replacing tabs at playground (#24285)
add 166b881cb4c [#24339] Make Slices use iterable coder instead of custom coder. (#24346)
add 1335b98ce54 Add custom inference fns to CHANGES.md (#24412)
add 7286f55fb5f Better warning and Exception message in CalciteUtil (#24414)
add 2fc974cce7b List breaking change #24339 in Changes.md (#24420)
add 9a575b742e1 Allow composite output types in sql.Transform. (#24421)
add 3951a7d0ad9 Add map_windows support to Go SDK (#24307)
add 58b340cd0f5 Fix dependencies of archetype tasks
add 596e19f938a Upgrade checker framework to 3.13.0
add 82150176099 Upgrade checker framework to 3.14.0
add 2af405f5ff2 Upgrade checker framework to 3.15.0
add d548c116f0e Merge pull request #24404: Upgrade checker framework to 3.15.0
add fccc46bc180 Inline :sdks:java:core:buildDependents so we can incrementally split
add 2f7cb556885 Merge pull request #24391: Inline :sdks:java:core:buildDependents so we can incrementally split
add 3bd76cc130f Deleted initialNumReaders paramter. (#24355)
add 466db663a9b [Website] update copy-to-clipboard.js #24372
add 4ab557bf987 Merge pull request #24370: [Website] update copy-to-clipboard.js
add fdc70cfdc51 Bump google.golang.org/grpc from 1.50.1 to 1.51.0 in /sdks (#24281)
add 58f7399ca24 [Playground] use JAVA SDK 2.43.0 in Examples CI (#24429)
add e32b05408c5 Update authors.yml (#24433)
add b8b4b6c9031 Bump cloud.google.com/go/spanner from 1.36.0 to 1.40.0 in /sdks (#24423)
add 050288545cc Add Large Language Model RunInference Example (#24350)
add 8802f277e34 [Playground] [Backend] minor fixes for error msgs (#23999)
add a997f9c28bb pg_24284_now_closing_parenthesis on cancel button is visible (#24327)
add 012c885af43 [Github Actions] - Cut Release Branch Workflow (#24020)
add 6be2eed4563 Add six to build-requirements.txt (#24434)
add 8a42014f9a4 Add Pytorch RunInference GPU benchmark (#24347)
add 9915ec466b2 Fix multiple mutations affecting the same entity in Datastore write (#24403)
add 8349ee6c12a Fix BlobstorageIO.checksum Attribute Error (#24442)
No new revisions were added by this update.
Summary of changes:
.github/workflows/cut_release_branch.yml | 111 +++++++
.github/workflows/dask_runner_tests.yml | 2 +-
.../workflows/playground_examples_ci_reusable.yml | 6 +-
.../workflows/tour_of_beam_backend_integration.yml | 2 +-
.github/workflows/tour_of_beam_examples_ci.yml | 6 +-
.test-infra/jenkins/LoadTestsBuilder.groovy | 15 +-
.test-infra/jenkins/PythonTestProperties.groovy | 1 +
.../job_InferenceBenchmarkTests_Python.groovy | 42 ++-
.../job_PostCommit_Python_Examples_Flink.groovy | 1 -
CHANGES.md | 6 +
CI.md | 6 +
build.gradle.kts | 133 +++++++-
buildSrc/build.gradle.kts | 2 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 102 +++----
.../datatokenization/DataTokenization.java | 2 +-
.../transforms/DataProtectors.java | 9 +-
.../datatokenization/transforms/JsonToBeamRow.java | 2 +-
.../datatokenization/utils/CsvConverters.java | 2 +-
.../datatokenization/utils/ErrorConverters.java | 4 +-
.../datatokenization/utils/SchemasUtils.java | 6 +-
learning/tour-of-beam/backend/README.md | 8 +-
learning/tour-of-beam/backend/function.go | 4 +-
.../backend/integration_tests/auth_emulator.go | 4 +-
.../backend/integration_tests/auth_test.go | 99 +++++-
.../backend/integration_tests/client_pg.go | 39 +++
learning/tour-of-beam/backend/internal/entity.go | 9 +-
.../backend/internal/persistence_key.go | 23 +-
.../backend/internal/service/content.go | 24 +-
.../helper.go => internal/service/mock_helper.go} | 20 +-
.../backend/internal/service/pg_adapter.go | 3 +-
.../backend/internal/storage/datastore.go | 31 +-
.../tour-of-beam/backend/internal/storage/iface.go | 6 +-
.../backend/internal/storage/index.yaml | 4 +
.../tour-of-beam/backend/internal/storage/mock.go | 5 +-
.../backend/internal/storage/schema.go | 7 +-
.../tour-of-beam/backend/playground_api/api.pb.go | 336 ++++++++++----------
playground/backend/cmd/server/controller.go | 124 ++++----
playground/backend/containers/java/Dockerfile | 2 +-
.../backend/internal/db/datastore/datastore_db.go | 4 +-
.../repositories/code_client/grpc_code_client.dart | 23 +-
.../example_client/grpc_example_client.dart | 13 +-
.../lib/src/util/replace_incorrect_symbols.dart | 24 --
.../lib/src/widgets/run_button.dart | 10 +-
playground/infrastructure/helper.py | 16 +-
.../dataflow/worker/WorkItemStatusClient.java | 25 +-
.../dataflow/worker/WorkItemStatusClientTest.java | 17 ++
runners/portability/java/build.gradle | 8 +-
.../translation/PipelineTranslator.java | 11 +-
.../translation/helpers/EncoderHelpers.java | 3 +-
sdks/go.mod | 12 +-
sdks/go.sum | 26 +-
sdks/go/pkg/beam/coder.go | 21 +-
sdks/go/pkg/beam/core/graph/coder/coder.go | 10 +
sdks/go/pkg/beam/core/runtime/exec/coder.go | 42 ++-
sdks/go/pkg/beam/core/runtime/exec/coder_test.go | 3 +
sdks/go/pkg/beam/core/runtime/exec/datasource.go | 2 +-
sdks/go/pkg/beam/core/runtime/exec/pardo.go | 2 +-
sdks/go/pkg/beam/core/runtime/exec/translate.go | 68 +++++
.../pkg/beam/core/runtime/exec/translate_test.go | 94 ++++++
sdks/go/pkg/beam/core/runtime/exec/window.go | 50 +++
sdks/go/pkg/beam/core/runtime/exec/window_test.go | 63 ++++
sdks/go/pkg/beam/core/runtime/graphx/coder.go | 51 ++--
sdks/go/pkg/beam/core/runtime/graphx/coder_test.go | 87 ++++--
sdks/go/pkg/beam/core/runtime/graphx/dataflow.go | 50 ++-
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 1 +
sdks/go/pkg/beam/core/typex/fulltype.go | 35 ++-
sdks/go/pkg/beam/core/typex/fulltype_test.go | 22 ++
sdks/go/pkg/beam/transforms/sql/sql.go | 14 +-
sdks/go/test/integration/integration.go | 2 -
.../container/license_scripts/dep_urls_java.yaml | 2 +-
.../org/apache/beam/sdk/io/LocalResourceId.java | 7 +-
.../beam/sdk/metrics/MetricsEnvironment.java | 2 +-
.../util/UnboundedScheduledExecutorService.java | 16 +-
.../service/JavaClassLookupTransformProvider.java | 6 +-
.../beam/sdk/extensions/python/PythonService.java | 2 +-
.../beam/sdk/extensions/sql/jdbc/BeamSqlLine.java | 2 +-
.../sdk/extensions/sql/jdbc/BeamSqlLineIT.java | 2 +-
.../extensions/sql/impl/rel/CalcRelSplitter.java | 2 +-
.../extensions/sql/impl/utils/CalciteUtils.java | 18 +-
.../provider/kafka/NestedPayloadKafkaTable.java | 2 +-
.../pubsublite/PubsubLiteTableProviderTest.java | 4 +-
.../zetasketch/ApproximateCountDistinct.java | 4 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 5 +-
.../sdk/fn/stream/BufferingStreamObserver.java | 4 +-
.../beam/sdk/fn/stream/PrefetchableIterables.java | 2 +-
sdks/java/harness/build.gradle | 71 +++--
sdks/java/harness/jmh/build.gradle | 1 +
.../fn/harness/jmh/CombinerTableBenchmark.java | 84 -----
.../jmh/PrecombineGroupingTableBenchmark.java | 112 +++++++
.../fn/harness/control/ExecutionStateSampler.java | 3 +-
.../sdk/io/aws/dynamodb/AwsClientsProvider.java | 2 +-
.../beam/sdk/io/aws/sns/AwsClientsProvider.java | 2 +-
.../beam/sdk/io/gcp/bigquery/AppendClientInfo.java | 68 +++++
.../io/gcp/bigquery/BeamRowToStorageApiProto.java | 144 ++++-----
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 12 +-
.../beam/sdk/io/gcp/bigquery/BigQueryOptions.java | 6 -
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 2 +-
.../sdk/io/gcp/bigquery/SplittingIterable.java | 31 +-
.../bigquery/StorageApiDynamicDestinations.java | 21 +-
.../StorageApiDynamicDestinationsBeamRow.java | 50 ++-
.../StorageApiDynamicDestinationsTableRow.java | 201 +++++-------
.../beam/sdk/io/gcp/bigquery/StorageApiLoads.java | 12 +-
.../io/gcp/bigquery/StorageApiWritePayload.java | 5 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 155 +++++-----
.../bigquery/StorageApiWritesShardedRecords.java | 214 +++++++------
.../io/gcp/bigquery/TableRowToStorageApiProto.java | 337 ++++++++++++++++-----
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 9 +-
.../sdk/io/gcp/firestore/FirestoreV1ReadFn.java | 2 +-
.../io/gcp/healthcare/FhirIOPatientEverything.java | 2 +-
.../beam/sdk/io/gcp/pubsub/NestedRowToMessage.java | 2 +-
.../internal/LimitingTopicBacklogReader.java | 4 +-
.../internal/MemoryBufferedSubscriberImpl.java | 2 +-
.../internal/TopicBacklogReaderImpl.java | 2 +-
.../changestreams/action/ActionFactory.java | 2 +-
.../changestreams/encoder/TimestampEncoding.java | 2 +-
.../changestreams/mapper/MapperFactory.java | 2 +-
.../model/ChangeStreamRecordMetadata.java | 5 +-
.../gcp/bigquery/BeamRowToStorageApiProtoTest.java | 28 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 273 ++++-------------
.../sdk/io/gcp/bigquery/BigQueryUtilsTest.java | 26 +-
.../bigquery/TableRowToStorageApiProtoTest.java | 82 ++++-
.../beam/sdk/io/gcp/datastore/DatastoreV1Test.java | 50 ++-
.../io/gcp/datastore/RampupThrottlingFnTest.java | 9 +-
.../internal/OffsetByteRangeTrackerTest.java | 4 +-
.../internal/PerSubscriptionPartitionSdfTest.java | 4 +-
.../beam/sdk/io/singlestore/SingleStoreIO.java | 41 +--
.../sdk/io/singlestore/ReadWithPartitionsTest.java | 76 +----
sdks/java/io/sparkreceiver/2/build.gradle | 1 +
.../sdk/io/sparkreceiver/SparkReceiverIOIT.java | 12 +-
sdks/java/maven-archetypes/examples/build.gradle | 1 +
.../maven-archetypes/gcp-bom-examples/build.gradle | 1 +
.../apache_beam/examples/complete/tfidf_it_test.py | 1 +
.../cookbook/bigquery_side_input_it_test.py | 1 +
.../examples/cookbook/mergecontacts_test.py | 1 +
.../cookbook/multiple_output_pardo_test.py | 1 +
.../inference/large_language_modeling/main.py | 140 +++++++++
.../large_language_modeling/requirements.txt | 10 +-
.../inference/pytorch_image_classification.py | 5 +-
.../kfp/components/preprocessing/requirements.txt | 2 +-
sdks/python/apache_beam/io/azure/blobstorageio.py | 2 +-
.../testing/benchmarks/inference/README.md | 52 +++-
.../pytorch_image_classification_benchmarks.py | 5 +-
.../apache_beam/transforms/combiners_test.py | 61 ----
sdks/python/apache_beam/transforms/core.py | 14 +-
sdks/python/build-requirements.txt | 5 +
.../tensorrt_runinference/tensor_rt.dockerfile | 4 +-
sdks/python/test-suites/portable/common.gradle | 3 -
website/www/site/assets/js/copy-to-clipboard.js | 2 +
website/www/site/assets/scss/_global.sass | 3 +
.../en/documentation/ml/large-language-modeling.md | 73 +++++
.../site/content/en/documentation/ml/overview.md | 3 +-
website/www/site/data/authors.yml | 6 +-
.../partials/section-menu/en/documentation.html | 1 +
153 files changed, 2928 insertions(+), 1739 deletions(-)
create mode 100644 .github/workflows/cut_release_branch.yml
create mode 100644 learning/tour-of-beam/backend/integration_tests/client_pg.go
copy sdks/go/pkg/beam/core/util/protox/protox.go => learning/tour-of-beam/backend/internal/persistence_key.go (66%)
rename learning/tour-of-beam/backend/{playground_api/helper.go => internal/service/mock_helper.go} (60%)
delete mode 100644 playground/frontend/playground_components/lib/src/util/replace_incorrect_symbols.dart
delete mode 100644 sdks/java/harness/jmh/src/main/java/org/apache/beam/fn/harness/jmh/CombinerTableBenchmark.java
create mode 100644 sdks/java/harness/jmh/src/main/java/org/apache/beam/fn/harness/jmh/PrecombineGroupingTableBenchmark.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AppendClientInfo.java
create mode 100644 sdks/python/apache_beam/examples/inference/large_language_modeling/main.py
copy website/notebooks/imports/license.md => sdks/python/apache_beam/examples/inference/large_language_modeling/requirements.txt (88%)
create mode 100644 website/www/site/content/en/documentation/ml/large-language-modeling.md