You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2023/11/29 16:59:27 UTC
(beam) branch dependabot/go_modules/sdks/cloud.google.com/go/storage-1.35.1 updated (fb8e5b6e746 -> eca01ce89b2)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/storage-1.35.1
in repository https://gitbox.apache.org/repos/asf/beam.git
discard fb8e5b6e746 Bump cloud.google.com/go/storage from 1.34.1 to 1.35.1 in /sdks
add b212e6c747e Add website reviewers (#29459)
add ba67aa1a9f3 Make BigtableChangeStreamIT less flaky by adding a short sleep (#29277)
add a4d61357cd8 Add 'mean' and 'sum of squared deviations' stats to histogram (#29445)
add 65ddc344955 Gcsio migration (#29360) (#29465)
add bb310e0b87a Document compact lifecycle method (#29466)
add 67dbdd196ab Upgrade Kafka read/write transforms without upgrading the pipeline (#29362)
add 2d5bb29c290 Website updates for the 2.52.0 release (#29331)
add 1b3c7b1f696 Fix finalize_release (#29391)
add 9ec75ed009e Update release guide with post rc edits (#29377)
add e299a4e7442 Bump google.golang.org/api from 0.150.0 to 0.151.0 in /sdks (#29462)
add c1b83d2365c Copy edit the Custom Remote Inference notebook (#29460)
add c4e211b000a Fix a bunch of no-op typos (#29294)
add bc0efc4b79e Fix file extension for pypi artifacts (#29496)
add aaadb7712ab Define PerStepNamespaceMetrics proto that will be used to send metric updates on the GetData heartbeat requests. (#29482)
add 916125076b1 Enable the useless-with-lock check (#29468)
add ae433a92de4 Add type hint changes to CHANGES.md (#29469)
add a22ecf2001e Bump github.com/nats-io/nats-server/v2 from 2.10.4 to 2.10.5 in /sdks (#29493)
add 3439d6ab259 Bump golang.org/x/net from 0.17.0 to 0.18.0 in /sdks (#29464)
add c8a6d4593e7 [RRIO][Python] Replace call with __call__ to make caller callable (#29497)
add b7acf575f6c Adding instructions for PMC finalization (#29483)
add 53ffdd74d25 Fix kafka emulator dist name (#29499)
add 39d3e365f19 remove unnecessary ObjectMapper (#29495)
add 32b10670c37 Bugfix/override table ref if null (#29471)
add 2fac45517cd Improve File IO Logging (#29485)
add 392a4a13842 Revert the value of MAX_BATCH_OPERATION_SIZE in gcsio (#29503)
add e47054ebd04 Simplify passing custom requirement files. (#29502)
add bb4b633b019 Fix BigQueryStorageWriteApiSchemaTransformProvider null pointer on numStreams (#29434)
add a349e765ff9 Collect metric of message size distribution in KafkaIO read. (#29443)
add 0972bc0170b Propagate correct boundedness when using multiple outputs (#29506)
add 444e10d29e0 Kafka adaptive timeout implementation to handle empty topic cases (#29400)
add 074a01ccfcf Update the About ML page and create a redirect from the Python page (#29509)
add 99314816b0f Add package-info to fix checkstyle validation (#29521)
add e863d919a32 Improve actionability on IT errors (with Dataflow job URL) (#29520)
add 09a2cfcc2b2 Bump grpc_version dependency (#29511)
add 23b1fe96128 add canonical link into javadocs when running post commit (#29475)
add 9ee6a012e03 Attach exact cause in RuntimeException raised (#29470)
add 00d5863551c add canonical link into javadocs when releasing javadoc (#29473)
add 1e06d882bdd [RRIO] [Throttle] [Cache] Implement Throttle and Cache using an external resource. (#29401)
add 200aa46b2a7 [prism] Fix downstream side input computations. (#29439)
add 40a9882343c Update Java dependency management page (#29513)
add 2a20b097262 Close the gRPC channel started for the expansion service (#29524)
add 183141d2eab revert protobuf version (#29523)
add 6ce104769c4 Change StateBackedIterable to implement ElementByteSizeObservableIterable avoiding iteration to estimate observe bytes. (#29517)
add b6e19eb85a7 Add bool type to ClickHouse SDK (#29522)
add 5a6329492f7 Revert "revert protobuf version (#29523)" (#29530)
add db1afc09613 Shutdown remaining Jenkins PreCommit and Performance Test jobs (#29534)
add 3e0d0d62a5b Clickhouse support low cardinality (#29533)
add 2f4c9029697 Fix rendering of playground deployment steps (#29519)
add 8830fab1c49 Respect useReflectApi parameter (#29518)
add 109133e38ed Remove SuppressWarnings nullness from JsonMatcher (#29486)
add db6a55bc93d Clarify description of MLTransform. (#29539)
add 86f711b1bf7 Don't pin an older commit in the documentation. (#29540)
add 29cd0aa8fe5 Add warning for unknown pipeline options (#29474)
add cbddd4cec3b Fix NPE due to racing condition in hadoop.SerializableConfiguration (#29537)
add df324fdc064 chore: clean up BigtableIO client creation code (#29538)
add d9523df08f7 Fix blog date format (#29544)
add f6842049064 Bump github.com/aws/aws-sdk-go-v2/config from 1.25.1 to 1.25.8 in /sdks (#29556)
add eca01ce89b2 Bump cloud.google.com/go/storage from 1.34.1 to 1.35.1 in /sdks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (fb8e5b6e746)
\
N -- N -- N refs/heads/dependabot/go_modules/sdks/cloud.google.com/go/storage-1.35.1 (eca01ce89b2)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/REVIEWERS.yml | 8 +
.github/workflows/README.md | 11 +
.github/workflows/beam_PostCommit_Javadoc.yml | 5 +
.github/workflows/build_release_candidate.yml | 5 +
.github/workflows/finalize_release.yml | 30 +-
.test-infra/jenkins/README.md | 147 +-----
.test-infra/jenkins/job_Java_Jmh.groovy | 49 --
.test-infra/jenkins/job_PreCommit_Java.groovy | 122 -----
.../job_PreCommit_Java_GCP_IO_Direct.groovy | 67 ---
.test-infra/jenkins/job_PreCommit_Java_IOs.groovy | 263 ----------
...Java_PortableValidatesRunner_Flink_Batch.groovy | 39 --
.test-infra/jenkins/job_PreCommit_Python.groovy | 42 --
.../jenkins/job_PreCommit_Python_Coverage.groovy | 40 --
.../jenkins/job_PreCommit_Python_Dataframes.groovy | 41 --
.../jenkins/job_PreCommit_Python_Examples.groovy | 41 --
.../jenkins/job_PreCommit_Python_Runners.groovy | 41 --
.../jenkins/job_PreCommit_Python_Transforms.groovy | 41 --
...b_PreCommit_Python_ValidatesRunner_Flink.groovy | 46 --
.../configmap.yaml | 30 ++
.../deployment.yaml | 27 +
.../kustomization.yaml | 31 +-
CHANGES.md | 4 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
contributor-docs/release-guide.md | 105 ++--
.../beam/examples/complete/AutoComplete.java | 6 +-
.../beam-ml/custom_remote_inference.ipynb | 52 +-
.../beam-ml/image_processing_tensorflow.ipynb | 2 +-
examples/notebooks/beam-ml/mltransform_basic.ipynb | 4 +-
.../beam-ml/nlp_tensorflow_streaming.ipynb | 2 +-
.../apache/beam/it/conditions/ConditionCheck.java | 2 +-
.../it/gcp/dataflow/AbstractPipelineLauncher.java | 5 +-
.../internal/environment/environment_service.go | 2 +-
playground/kafka-emulator/build.gradle | 4 +-
.../core/construction/TransformUpgrader.java | 49 +-
.../unbounded/FlinkUnboundedSourceReader.java | 3 +
.../streaming/stableinput/BufferingDoFnRunner.java | 2 +-
.../beam/runners/dataflow/DataflowRunner.java | 2 +-
.../dataflow/options/DataflowPipelineOptions.java | 2 +-
.../runners/dataflow/util/RandomAccessData.java | 6 +-
.../beam/runners/dataflow/util/TimeUtil.java | 2 +-
.../worker/DataflowExecutionStateRegistry.java | 2 +-
.../worker/windmill/src/main/proto/windmill.proto | 75 +++
.../artifact/ArtifactStagingService.java | 6 +-
sdks/go.mod | 34 +-
sdks/go.sum | 68 +--
sdks/go/examples/snippets/04transforms.go | 5 +
sdks/go/pkg/beam/io/fileio/read_test.go | 16 +-
.../prism/internal/engine/elementmanager.go | 39 +-
sdks/go/pkg/beam/runners/prism/internal/execute.go | 29 +-
.../beam/runners/prism/internal/handlecombine.go | 6 +-
.../runners/prism/internal/handlecombine_test.go | 212 ++++++++
.../pkg/beam/runners/prism/internal/handlepardo.go | 2 +-
.../pkg/beam/runners/prism/internal/preprocess.go | 180 ++++---
.../beam/runners/prism/internal/preprocess_test.go | 504 +++++++++++++++++-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 18 +-
sdks/go/pkg/beam/util/pubsubx/pubsub_test.go | 4 +
.../java/org/apache/beam/agent/package-info.java | 20 +-
.../org/apache/beam/sdk/coders/AtomicCoder.java | 2 +-
.../apache/beam/sdk/coders/IterableLikeCoder.java | 2 +-
.../apache/beam/sdk/coders/RowCoderGenerator.java | 2 +-
.../java/org/apache/beam/sdk/io/FileBasedSink.java | 2 +-
.../org/apache/beam/sdk/testing/JsonMatcher.java | 13 +-
.../java/org/apache/beam/sdk/testing/PAssert.java | 2 +-
.../java/org/apache/beam/sdk/transforms/Regex.java | 2 +-
.../java/org/apache/beam/sdk/transforms/Sets.java | 2 +-
.../reflect/ByteBuddyDoFnInvokerFactory.java | 2 +-
.../java/org/apache/beam/sdk/util/ApiSurface.java | 8 +-
.../org/apache/beam/sdk/util/HistogramData.java | 41 ++
.../beam/sdk/util/common/ReflectHelpers.java | 2 +-
.../sdk/io/BoundedReadFromUnboundedSourceTest.java | 2 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 10 +-
.../beam/sdk/transforms/windowing/WindowTest.java | 2 +-
.../apache/beam/sdk/util/HistogramDataTest.java | 22 +
.../beam/sdk/util/common/ReflectHelpersTest.java | 4 +-
.../sdk/expansion/service/ExpansionService.java | 10 +-
.../beam/sdk/extensions/avro/coders/AvroCoder.java | 2 +-
.../core/annotation/operator/package-info.java | 2 +-
.../core/client/operator/ReduceWindow.java | 2 +-
.../core/client/operator/base/Builders.java | 2 +-
.../euphoria/core/translate/JoinTranslator.java | 2 +-
.../provider/GenericTranslatorProvider.java | 2 +-
.../core/docs/DocumentationExamplesTest.java | 12 +-
.../beam/sdk/extensions/sql/impl/JdbcDriver.java | 3 -
.../sql/impl/rel/BeamLogicalConvention.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamUnnestRel.java | 2 +-
.../udf/BeamSalUhfSpecialTypeAndValueTest.java | 6 +-
.../provider/bigtable/BigtableTableFlatTest.java | 2 +-
.../sql/meta/provider/kafka/KafkaTestTable.java | 8 +-
.../beam/fn/harness/state/StateBackedIterable.java | 87 +++-
.../fn/harness/state/StateBackedIterableTest.java | 58 +++
.../beam/sdk/io/clickhouse/ClickHouseIO.java | 4 +-
.../beam/sdk/io/clickhouse/ClickHouseWriter.java | 3 +
.../apache/beam/sdk/io/clickhouse/TableSchema.java | 11 +-
.../clickhouse/src/main/javacc/ColumnTypeParser.jj | 62 ++-
.../beam/sdk/io/clickhouse/ClickHouseIOTest.java | 20 +-
.../beam/sdk/io/clickhouse/TableSchemaTest.java | 5 +
.../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 2 +-
sdks/java/io/expansion-service/build.gradle | 2 +
.../beam/sdk/io/gcp/bigquery/AppendClientInfo.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 4 +-
.../sdk/io/gcp/bigquery/BigQueryIOMetadata.java | 2 +-
.../io/gcp/bigquery/BigQueryResourceNaming.java | 2 +-
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 20 +-
.../beam/sdk/io/gcp/bigquery/RetryManager.java | 2 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 12 +-
...ueryStorageWriteApiSchemaTransformProvider.java | 9 +-
.../io/gcp/bigtable/BigtableConfigTranslator.java | 7 +
.../io/gcp/bigtable/BigtableServiceFactory.java | 7 +-
.../sdk/io/gcp/bigtable/BigtableServiceImpl.java | 24 +-
.../io/gcp/bigquery/BigQueryServicesImplTest.java | 35 ++
.../changestreams/it/BigtableChangeStreamIT.java | 35 +-
...mplTest.java => BlockingCommitterImplTest.java} | 2 +-
.../sdk/io/hadoop/SerializableConfiguration.java | 23 +-
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 85 +--
.../org/apache/beam/sdk/io/kafka/KafkaIOUtils.java | 4 +-
.../beam/sdk/io/kafka/KafkaUnboundedReader.java | 27 +-
.../beam/sdk/io/kafka/ReadFromKafkaDoFn.java | 11 +
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 62 +++
.../beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java | 31 ++
sdks/java/io/kafka/upgrade/build.gradle | 46 ++
.../sdk/io/kafka/upgrade/KafkaIOTranslation.java | 578 +++++++++++++++++++++
.../beam/sdk/io/kafka/upgrade/package-info.java | 22 +-
.../io/kafka/upgrade/KafkaIOTranslationTest.java | 256 +++++++++
sdks/java/io/rrio/build.gradle | 1 +
.../org/apache/beam/io/requestresponse/Cache.java | 239 +++++++++
.../apache/beam/io/requestresponse/CacheRead.java | 121 -----
.../apache/beam/io/requestresponse/CacheWrite.java | 119 -----
.../org/apache/beam/io/requestresponse/Call.java | 39 +-
.../org/apache/beam/io/requestresponse/Quota.java | 70 +++
.../beam/io/requestresponse/RedisClient.java | 10 +
.../beam/io/requestresponse/ThrottleDequeue.java | 101 ----
.../beam/io/requestresponse/ThrottleEnqueue.java | 61 ---
.../io/requestresponse/ThrottleRefreshQuota.java | 55 --
.../ThrottleWithExternalResource.java | 418 +++++++++++++++
.../apache/beam/io/requestresponse/CacheIT.java | 120 +++++
.../apache/beam/io/requestresponse/CacheTest.java | 132 +++++
.../apache/beam/io/requestresponse/CallTest.java | 112 +++-
...java => EchoGRPCCallerWithSetupTeardownIT.java} | 14 +-
...HTTPCallerTestIT.java => EchoHTTPCallerIT.java} | 18 +-
.../beam/io/requestresponse/EchoITOptions.java | 7 +-
.../beam/io/requestresponse/EchoRequestCoder.java | 44 ++
.../{RedisClientTestIT.java => RedisClientIT.java} | 24 +-
.../ThrottleWithExternalResourceIT.java | 186 +++++++
.../ThrottleWithExternalResourceTest.java | 77 +++
sdks/python/.pylintrc | 1 -
.../apache_beam/examples/snippets/snippets_test.py | 4 +
sdks/python/apache_beam/io/fileio.py | 4 +-
.../apache_beam/io/gcp/bigquery_avro_tools.py | 2 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 2 +-
sdks/python/apache_beam/io/requestresponseio.py | 4 +-
.../apache_beam/io/requestresponseio_it_test.py | 14 +-
sdks/python/apache_beam/ml/inference/base.py | 3 +-
.../python/apache_beam/options/pipeline_options.py | 6 +-
sdks/python/apache_beam/pvalue.py | 7 +-
.../apache_beam/testing/analyzers/__init__.py | 2 +-
sdks/python/apache_beam/transforms/core_test.py | 28 +
sdks/python/apache_beam/utils/subprocess_server.py | 15 +-
sdks/python/scripts/run_integration_test.sh | 16 +-
settings.gradle.kts | 1 +
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.52.0.md | 137 +++++
.../site/content/en/documentation/ml/about-ml.md | 341 +++++++++++-
.../content/en/documentation/programming-guide.md | 23 +-
.../en/documentation/sdks/java-dependencies.md | 138 ++++-
.../documentation/sdks/python-machine-learning.md | 435 ----------------
.../transforms/python/elementwise/mltransform.md | 2 +-
.../www/site/content/en/get-started/downloads.md | 14 +-
167 files changed, 4978 insertions(+), 2532 deletions(-)
delete mode 100644 .test-infra/jenkins/job_Java_Jmh.groovy
delete mode 100644 .test-infra/jenkins/job_PreCommit_Java.groovy
delete mode 100644 .test-infra/jenkins/job_PreCommit_Java_GCP_IO_Direct.groovy
delete mode 100644 .test-infra/jenkins/job_PreCommit_Java_IOs.groovy
delete mode 100644 .test-infra/jenkins/job_PreCommit_Java_PortableValidatesRunner_Flink_Batch.groovy
delete mode 100644 .test-infra/jenkins/job_PreCommit_Python.groovy
delete mode 100644 .test-infra/jenkins/job_PreCommit_Python_Coverage.groovy
delete mode 100644 .test-infra/jenkins/job_PreCommit_Python_Dataframes.groovy
delete mode 100644 .test-infra/jenkins/job_PreCommit_Python_Examples.groovy
delete mode 100644 .test-infra/jenkins/job_PreCommit_Python_Runners.groovy
delete mode 100644 .test-infra/jenkins/job_PreCommit_Python_Transforms.groovy
delete mode 100644 .test-infra/jenkins/job_PreCommit_Python_ValidatesRunner_Flink.groovy
create mode 100644 .test-infra/mock-apis/infrastructure/kubernetes/refresher/overlays/echo-ThrottleWithExternalResourceTestIT-10-per-1s-quota/configmap.yaml
create mode 100644 .test-infra/mock-apis/infrastructure/kubernetes/refresher/overlays/echo-ThrottleWithExternalResourceTestIT-10-per-1s-quota/deployment.yaml
copy sdks/python/apache_beam/testing/analyzers/__init__.py => .test-infra/mock-apis/infrastructure/kubernetes/refresher/overlays/echo-ThrottleWithExternalResourceTestIT-10-per-1s-quota/kustomization.yaml (50%)
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/handlecombine_test.go
rename .test-infra/jenkins/job_PreCommit_Python_DockerBuild.groovy => sdks/java/container/agent/src/main/java/org/apache/beam/agent/package-info.java (69%)
rename sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/internal/{BlockingCommmitterImplTest.java => BlockingCommitterImplTest.java} (98%)
create mode 100644 sdks/java/io/kafka/upgrade/build.gradle
create mode 100644 sdks/java/io/kafka/upgrade/src/main/java/org/apache/beam/sdk/io/kafka/upgrade/KafkaIOTranslation.java
rename .test-infra/jenkins/job_PreCommit_PythonIT.groovy => sdks/java/io/kafka/upgrade/src/main/java/org/apache/beam/sdk/io/kafka/upgrade/package-info.java (63%)
create mode 100644 sdks/java/io/kafka/upgrade/src/test/java/org/apache/beam/sdk/io/kafka/upgrade/KafkaIOTranslationTest.java
create mode 100644 sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/Cache.java
delete mode 100644 sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/CacheRead.java
delete mode 100644 sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/CacheWrite.java
create mode 100644 sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/Quota.java
delete mode 100644 sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/ThrottleDequeue.java
delete mode 100644 sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/ThrottleEnqueue.java
delete mode 100644 sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/ThrottleRefreshQuota.java
create mode 100644 sdks/java/io/rrio/src/main/java/org/apache/beam/io/requestresponse/ThrottleWithExternalResource.java
create mode 100644 sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/CacheIT.java
create mode 100644 sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/CacheTest.java
rename sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/{EchoGRPCCallerWithSetupTeardownTestIT.java => EchoGRPCCallerWithSetupTeardownIT.java} (91%)
rename sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/{EchoHTTPCallerTestIT.java => EchoHTTPCallerIT.java} (87%)
create mode 100644 sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/EchoRequestCoder.java
rename sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/{RedisClientTestIT.java => RedisClientIT.java} (89%)
create mode 100644 sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/ThrottleWithExternalResourceIT.java
create mode 100644 sdks/java/io/rrio/src/test/java/org/apache/beam/io/requestresponse/ThrottleWithExternalResourceTest.java
create mode 100644 website/www/site/content/en/blog/beam-2.52.0.md
delete mode 100644 website/www/site/content/en/documentation/sdks/python-machine-learning.md