You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2025/02/04 20:57:57 UTC
(beam) branch dependabot/go_modules/sdks/google.golang.org/api-0.219.0 updated (da198d3ddff -> fd6d1b2056c)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/go_modules/sdks/google.golang.org/api-0.219.0
in repository https://gitbox.apache.org/repos/asf/beam.git
discard da198d3ddff Bump google.golang.org/api from 0.218.0 to 0.219.0 in /sdks
add 6ea382827eb Improve global sequence processing in Java SDK's "ordered" extension (#33629)
add 41cda9ff7d8 Skip highly flaky test (#33804)
add 3d4615e2fe9 support dbcp2.BasicDataSource in jdbcio lineage (#33801)
add 5da82e7f307 Revert "Increase timeouts in CallTest to reduce flakiness (#33775)" (#33807)
add fd5e212737a Fix PostRelease Nightly Snapshot (#33798)
add d121703693d Fix build-wheels python workflow (#33808)
add 4a09fc4481f Fix PreCommit Java IOs Direct job (#33813)
add fbfad87e845 Increase timeout for python performance test gradle task (#33797)
add 24827ebac37 Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#33812)
add 56f6e6aee50 Change Reify.asIterable to GBK in BigQueryIO File loads (#33587)
add 5da70fdf545 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#33810)
add 1742b2ecede Throw exception if Dataflow template file writing fails. (#33684)
add bbe63947aa6 Report delta StringSet counters for streaming (#33691)
add ebd8898e054 Run a significantly smaller Dask Graph. (#33806)
add 23607117f35 SnowflakeIO: do not quote empty fields (#33803)
add ea69d81747c Revert "Pause delete images in public AR (#32354)" (#33749)
add 7973577bc61 Fix failing extractMetricUpdatesStringSet (#33823)
add 6e3cf2b1130 Added 2025.md to track Beam docs (#33824)
add 332122410db Add python ml extras (#33825)
add 0ba6c8ab37b Introduce a BoundedTrie metric which is used to efficiently store and aggregate a collection of string sequences (FQNs) with a limited size.
add b47646643a3 Address comments, perform deep copies and support synchronization for mutable BoundedTrieData
add 7849f5fceef Add more unit tests for trienode and fix merge on empty node
add aaf3bd89877 Add more unit test for BoundedTrieData which are not already covered in fuzzy run tests
add 2b41b17cd90 Add unit tests for Cell
add 631a34bdc24 Plumb BoundedTrie to MetricsContainerImpl
add ba85636bc12 Plumb BoundedTrie to StreamingStepMetricsContainer
add a60330f0c04 Plumb BoundedTrie to StreamingStepMetricsContainer
add 5e2ec9e0e9e Plumb BoundedTrie to MetricQueryResults
add b7b74e6b964 Plumb BoundedTrie to MetricsContainerStepMap and make combine return reference
add 6cef4c5f1df Add test in MetricsContainerStepMapTest
add 6cd8fdcf302 Plumb BoundedTrie to DirectMetrics
add a7e237ef0ad Add BoundedTrieResult class
add 4825bf24646 Plumb BoundedTrieResult and BoundedTrie in DataflowMetrics
add 59088792c69 Plumb BoundedTrie JetMetric and PortableMetric
add 78508bf31ae Add test for BoundedTrieResult and some more plumbing
add c959101476c Fixes for github checks
add 3761e595738 Multi-threaded test in MetricsContainerImplTest and some clean up
add 35ea66b09b2 Address comments part 2
add b9d39867a6c Drop support DataflowMetrics and MetricsToCounterUpdateConverter until Dataflow java client support is ready
add 31b41b04c70 rebase fixes
add f68fe7511f2 Suppress IS2_INCONSISTENT_SYNC warning
add 5e031abd232 Merge pull request #33385 Introduce a BoundedTrie metric.
add b1d5e000aa5 Upgrade calcite to 1.37.0 used by io-expansion service (#33826)
add bfa0c59ebcd The Bag Partition is now configurable. (#33805)
add b701737261e increase timeout (#33835)
add e380edf1974 Change runner to ubuntu-22.04 (#33816)
add 83b373570fe [flink-runner] Improve Datastream for batch performances (#32440)
add d63b8fb760e Fix Playground CI Nightly job (#33837)
add df13ffe96d6 sdks/python: enable named aggregation to deferred DataFrame groupby (#33672)
add c7b2695f364 Adding handling of Nulled lists to beam_row_from_dict (#33830)
add edf7c9067cc fix activate (#33839)
add 8cbccc9145d Kafka source offset-based deduplication. (#33596)
add 4d7802877ea [Prism] Disable trie tests. (#33843)
add 0230b5d420f Update Dataflow api client version (#33829)
add 5aae10db7af Add a flag to disable BoundedTrie metrics in Beam
add 8805ab2240c Merge pull request #33457 from rohitsinha54/btrie-disable-flag
add 908a50ba838 [Dataflow Streaming] Enable Windmill GetWork Response Batching by default (#33847)
add 7356785e9ba Calculate byte size via sampling in StateBackedIterable if size is not cheap to calculate (#33780)
add 906474398ff Revert "[Flink] disable operator chaining in validatesRunner" (#33838)
add 7bec450c0fb [Dataflow Streaming] Optimize failed key processing by indexing workitems by sharding key (#33755)
add 6ed460a5377 Bump beam-master container version (#33848)
add 14df78cb9c3 Write how-to doc on dataflow cost benchmarking (#33702)
add 2712794a3eb [Go SDK + Protos] Fix Proto Spec for Pane encoding + Go SDK implementation. (#33840)
add b69b7030458 [#31438] Trigger Precusor work for Prism. (#33763)
add a4d6487a407 Fix Postcommit Python Arm workflow (#33849)
add f7c7ea760af Bump cloud.google.com/go/spanner from 1.73.0 to 1.75.0 in /sdks (#33846)
add fd6d1b2056c Bump google.golang.org/api from 0.218.0 to 0.219.0 in /sdks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (da198d3ddff)
\
N -- N -- N refs/heads/dependabot/go_modules/sdks/google.golang.org/api-0.219.0 (fd6d1b2056c)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../IO_Iceberg_Integration_Tests.json | 2 +-
.../trigger_files/beam_PostCommit_Go_VR_Flink.json | 1 +
.../beam_PostCommit_Java_DataflowV1.json | 3 +-
.../beam_PostCommit_Java_Examples_Flink.json | 3 +-
.../beam_PostCommit_Java_PVR_Flink_Batch.json | 1 +
.../beam_PostCommit_Java_PVR_Flink_Docker.json | 2 +
.../beam_PostCommit_Java_PVR_Flink_Streaming.json | 3 +-
...beam_PostCommit_Java_ValidatesRunner_Flink.json | 1 +
.github/trigger_files/beam_PostCommit_Python.json | 1 +
.../trigger_files/beam_PostCommit_XVR_Flink.json | 3 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 8 +-
.../workflows/beam_Publish_Beam_SDK_Snapshots.yml | 2 +-
.github/workflows/build_wheels.yml | 2 +-
.../tools/stale_dataflow_prebuilt_image_cleaner.sh | 2 +-
CHANGES.md | 10 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
.../discussion-docs/2025.md | 15 +-
.../complete/game/utils/WriteToBigQuery.java | 4 +-
.../katas/coretransforms/flattenWith/Task.java | 2 +-
.../learning/katas/coretransforms/tee/Task.java | 13 +
.../FlattenWith/FlattenWith/task.py | 28 +-
.../katas/python/Core Transforms/Tee/Tee/task.py | 24 +-
.../beam/model/pipeline/v1/beam_runner_api.proto | 21 +-
playground/categories.yaml | 1 +
.../main/groovy/mobilegaming-java-dataflow.groovy | 21 +-
.../main/groovy/mobilegaming-java-direct.groovy | 17 +-
.../core/GroupAlsoByWindowViaWindowSetNewDoFn.java | 6 +-
.../{CounterCell.java => BoundedTrieCell.java} | 89 +-
.../beam/runners/core/metrics/BoundedTrieData.java | 598 ++++++++++++
.../runners/core/metrics/DefaultMetricResults.java | 10 +-
.../beam/runners/core/metrics/MetricUpdates.java | 11 +-
.../runners/core/metrics/MetricsContainerImpl.java | 108 ++-
.../core/metrics/MetricsContainerStepMap.java | 8 +
.../core/metrics/MonitoringInfoConstants.java | 5 +
.../core/metrics/MonitoringInfoEncodings.java | 16 +
.../core/metrics/SimpleMonitoringInfoBuilder.java | 11 +
.../beam/runners/core/metrics/StringSetCell.java | 8 +
.../runners/core/metrics/BoundedTrieCellTest.java | 162 ++++
.../runners/core/metrics/BoundedTrieNodeTest.java | 1017 ++++++++++++++++++++
.../core/metrics/MetricsContainerImplTest.java | 102 ++
.../core/metrics/MetricsContainerStepMapTest.java | 100 +-
.../core/metrics/MonitoringInfoEncodingsTest.java | 38 +
.../apache/beam/runners/direct/DirectMetrics.java | 47 +-
.../beam/runners/direct/DirectMetricsTest.java | 29 +-
.../metrics/CustomMetricQueryResults.java | 11 +
.../extensions/metrics/MetricsHttpSinkTest.java | 9 +-
.../translation/types/CoderTypeSerializer.java | 24 +-
runners/flink/flink_runner.gradle | 10 +-
.../runners/flink/FlinkExecutionEnvironments.java | 10 +
.../beam/runners/flink/FlinkPipelineOptions.java | 9 +-
.../FlinkStreamingAggregationsTranslators.java | 544 +++++++++++
.../flink/FlinkStreamingPipelineTranslator.java | 5 +-
.../FlinkStreamingPortablePipelineTranslator.java | 35 +-
.../flink/FlinkStreamingTransformTranslators.java | 441 +++------
.../beam/runners/flink/adapter/FlinkKey.java | 87 ++
.../wrappers/streaming/DoFnOperator.java | 118 ++-
.../streaming/ExecutableStageDoFnOperator.java | 81 +-
...ySelector.java => KvToFlinkKeyKeySelector.java} | 23 +-
.../streaming/PartialReduceBundleOperator.java | 181 ++++
...eySelector.java => SdfFlinkKeyKeySelector.java} | 27 +-
.../wrappers/streaming/SplittableDoFnOperator.java | 5 +-
.../wrappers/streaming/WindowDoFnOperator.java | 22 +-
.../wrappers/streaming/WorkItemKeySelector.java | 20 +-
.../wrappers/streaming/io/DedupingOperator.java | 8 +-
.../wrappers/streaming/io/source/FlinkSource.java | 27 +-
...or.java => LazyFlinkSourceSplitEnumerator.java} | 145 ++-
.../source/bounded/FlinkBoundedSourceReader.java | 6 +
.../streaming/state/FlinkStateInternals.java | 450 ++++++---
.../runners/flink/FlinkPipelineOptionsTest.java | 6 +-
.../beam/runners/flink/FlinkSubmissionTest.java | 48 +-
.../beam/runners/flink/adapter/FlinkKeyTest.java | 94 ++
.../flink/streaming/FlinkStateInternalsTest.java | 30 +-
.../wrappers/streaming/DoFnOperatorTest.java | 206 ++--
.../streaming/ExecutableStageDoFnOperatorTest.java | 34 +-
.../wrappers/streaming/WindowDoFnOperatorTest.java | 49 +-
runners/google-cloud-dataflow-java/build.gradle | 6 +-
.../beam/runners/dataflow/DataflowMetrics.java | 7 +-
.../beam/runners/dataflow/DataflowRunner.java | 27 +-
.../options/DataflowStreamingPipelineOptions.java | 1 +
.../beam/runners/dataflow/DataflowMetricsTest.java | 1 +
.../beam/runners/dataflow/DataflowRunnerTest.java | 228 +++--
.../options/DataflowPipelineOptionsTest.java | 88 +-
.../dataflow/worker/BatchModeExecutionContext.java | 2 +-
.../dataflow/worker/DataflowMetricsContainer.java | 6 +
.../worker/MetricsToCounterUpdateConverter.java | 5 +-
.../worker/StreamingStepMetricsContainer.java | 27 +-
.../dataflow/worker/streaming/ActiveWorkState.java | 141 ++-
.../worker/streaming/ComputationState.java | 8 +-
.../streaming/WorkHeartbeatResponseProcessor.java | 16 +-
.../worker/streaming/WorkIdWithShardingKey.java | 15 +-
.../worker/BatchModeExecutionContextTest.java | 2 +-
.../worker/StreamingDataflowWorkerTest.java | 117 ++-
.../worker/StreamingStepMetricsContainerTest.java | 8 +-
.../worker/streaming/ActiveWorkStateTest.java | 170 +++-
.../client/grpc/GrpcGetDataStreamTest.java | 2 +
.../runners/jet/FailedRunningPipelineResults.java | 6 +
.../{CounterImpl.java => BoundedTrieImpl.java} | 36 +-
.../beam/runners/jet/metrics/JetMetricResults.java | 50 +-
.../runners/jet/metrics/JetMetricsContainer.java | 25 +-
.../beam/runners/portability/PortableMetrics.java | 44 +-
.../runners/portability/PortableRunnerTest.java | 15 +
runners/prism/java/build.gradle | 3 +
sdks/go.mod | 22 +-
sdks/go.sum | 47 +-
sdks/go/pkg/beam/core/graph/coder/panes.go | 8 +-
sdks/go/pkg/beam/core/graph/coder/panes_test.go | 54 +-
.../pkg/beam/core/graph/window/trigger/trigger.go | 2 +-
sdks/go/pkg/beam/core/typex/special.go | 13 +
.../pkg/beam/runners/prism/internal/engine/data.go | 19 +-
.../prism/internal/engine/elementmanager.go | 20 +-
.../beam/runners/prism/internal/engine/strategy.go | 539 ++++++++++-
.../runners/prism/internal/engine/strategy_test.go | 388 ++++++++
sdks/go/pkg/beam/runners/prism/internal/execute.go | 7 +-
.../container/license_scripts/license_script.sh | 8 +-
.../org/apache/beam/sdk/metrics/BoundedTrie.java | 54 ++
.../apache/beam/sdk/metrics/BoundedTrieResult.java | 57 ++
.../beam/sdk/metrics/MetricQueryResults.java | 10 +-
.../java/org/apache/beam/sdk/metrics/Metrics.java | 61 +-
.../apache/beam/sdk/metrics/MetricsContainer.java | 6 +
...ugeMetrics.java => UsesBoundedTrieMetrics.java} | 7 +-
.../beam/sdk/metrics/BoundedTrieResultTest.java | 65 ++
.../org/apache/beam/sdk/metrics/MetricsTest.java | 67 +-
.../ordered/ContiguousSequenceRange.java | 38 +-
.../extensions/ordered/GlobalSequenceTracker.java | 23 +-
.../ordered/GlobalSequencesProcessorDoFn.java | 49 +-
.../extensions/ordered/OrderedEventProcessor.java | 6 +-
.../ordered/OrderedEventProcessorResult.java | 12 +-
.../ordered/OrderedProcessingHandler.java | 12 +-
.../ordered/OrderedProcessingStatus.java | 9 +-
.../extensions/ordered/PerKeyTickerGenerator.java | 40 +-
.../sdk/extensions/ordered/ProcessingState.java | 3 +-
.../beam/sdk/extensions/ordered/ProcessorDoFn.java | 3 +-
.../ordered/combiner/DefaultSequenceCombiner.java | 18 +-
.../ordered/combiner/SequenceRangeAccumulator.java | 3 +-
.../ordered/ContiguousSequenceRangeTest.java | 71 ++
.../OrderedEventProcessorPerKeySequenceTest.java | 135 ++-
.../ordered/OrderedEventProcessorTestBase.java | 8 +-
.../combiner/SequenceRangeAccumulatorTest.java | 62 ++
.../fn/harness/control/ExecutionStateSampler.java | 9 +
.../beam/fn/harness/state/StateBackedIterable.java | 42 +-
.../harness/control/ExecutionStateSamplerTest.java | 23 +
.../fn/harness/state/StateBackedIterableTest.java | 4 +-
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 11 +-
.../beam/sdk/io/gcp/bigquery/BigQueryOptions.java | 16 +
.../sdk/io/gcp/bigquery/CombineAsIterable.java | 37 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 23 +-
sdks/java/io/iceberg/hive/build.gradle | 19 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java | 54 +-
.../org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java | 15 +
.../beam/sdk/io/kafka/KafkaCheckpointMark.java | 21 +
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 53 +-
.../KafkaIOReadImplementationCompatibility.java | 1 +
.../org/apache/beam/sdk/io/kafka/KafkaIOUtils.java | 13 +
.../beam/sdk/io/kafka/KafkaUnboundedReader.java | 33 +-
.../beam/sdk/io/kafka/KafkaUnboundedSource.java | 23 +-
.../beam/sdk/io/kafka/KafkaIOExternalTest.java | 1 -
...KafkaIOReadImplementationCompatibilityTest.java | 15 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 61 +-
.../sdk/io/kafka/upgrade/KafkaIOTranslation.java | 10 +
.../io/kafka/upgrade/KafkaIOTranslationTest.java | 1 +
.../org/apache/beam/sdk/io/mqtt/MqttIOTest.java | 2 +-
.../apache/beam/io/requestresponse/CallTest.java | 18 +-
.../apache/beam/sdk/io/snowflake/SnowflakeIO.java | 16 +-
.../test/unit/write/SnowflakeIOWriteTest.java | 3 +-
sdks/python/apache_beam/dataframe/frames.py | 95 +-
sdks/python/apache_beam/dataframe/frames_test.py | 10 +
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 6 +-
.../apache_beam/io/gcp/bigquery_tools_test.py | 11 +-
.../io/gcp/healthcare/dicomio_integration_test.py | 11 +-
.../python/apache_beam/runners/dask/dask_runner.py | 50 +-
.../apache_beam/runners/dask/dask_runner_test.py | 19 +
.../runners/dask/transform_evaluator.py | 32 +-
.../apache_beam/testing/benchmarks/README.md | 122 +++
sdks/python/container/run_validatescontainer.sh | 2 +-
sdks/python/setup.py | 35 +-
sdks/python/test-suites/dataflow/common.gradle | 2 +-
sdks/python/test-suites/tox/py39/build.gradle | 4 +
sdks/python/tox.ini | 6 +-
.../en/documentation/ml/ml-dependency-extras.md | 41 +
.../site/content/en/documentation/ml/overview.md | 1 +
.../partials/section-menu/en/documentation.html | 1 +
.../shortcodes/flink_java_pipeline_options.html | 5 +
.../shortcodes/flink_python_pipeline_options.html | 5 +
183 files changed, 7614 insertions(+), 1692 deletions(-)
copy learning/tour-of-beam/learning-content/cross-language/multi-pipeline/python-example/input.txt => contributor-docs/discussion-docs/2025.md (51%)
copy runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/{CounterCell.java => BoundedTrieCell.java} (52%)
create mode 100644 runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/BoundedTrieData.java
create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/BoundedTrieCellTest.java
create mode 100644 runners/core-java/src/test/java/org/apache/beam/runners/core/metrics/BoundedTrieNodeTest.java
create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingAggregationsTranslators.java
create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/adapter/FlinkKey.java
rename runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/{KvToByteBufferKeySelector.java => KvToFlinkKeyKeySelector.java} (62%)
create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/PartialReduceBundleOperator.java
rename runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/{SdfByteBufferKeySelector.java => SdfFlinkKeyKeySelector.java} (65%)
copy runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/source/{FlinkSourceSplitEnumerator.java => LazyFlinkSourceSplitEnumerator.java} (55%)
create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/adapter/FlinkKeyTest.java
copy sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/Dependency.java => runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/WorkIdWithShardingKey.java (72%)
copy runners/jet/src/main/java/org/apache/beam/runners/jet/metrics/{CounterImpl.java => BoundedTrieImpl.java} (60%)
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/BoundedTrie.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/BoundedTrieResult.java
copy sdks/java/core/src/main/java/org/apache/beam/sdk/testing/{UsesGaugeMetrics.java => UsesBoundedTrieMetrics.java} (84%)
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/metrics/BoundedTrieResultTest.java
create mode 100644 sdks/java/extensions/ordered/src/test/java/org/apache/beam/sdk/extensions/ordered/ContiguousSequenceRangeTest.java
copy examples/multi-language/src/main/java/org/apache/beam/examples/multilanguage/JavaPrefix.java => sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/CombineAsIterable.java (56%)
create mode 100644 sdks/python/apache_beam/testing/benchmarks/README.md
create mode 100644 website/www/site/content/en/documentation/ml/ml-dependency-extras.md