You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2022/07/22 16:21:12 UTC
[beam] branch dependabot/go_modules/sdks/cloud.google.com/go/bigquery-1.36.0 updated (4023e7336fb -> 35162273cf4)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/bigquery-1.36.0
in repository https://gitbox.apache.org/repos/asf/beam.git
discard 4023e7336fb Bump cloud.google.com/go/bigquery from 1.35.0 to 1.36.0 in /sdks
add 2eaab62373d Support combiner lifting.
add 12754bbe42b More clarification.
add 9697c13d06e Merge pull request #22330 Support combiner lifting.
add caaefc0c0b6 [CdapIO] HasOffset interface was implemented (#22193)
add 50346b5d141 added olehborysevych as collaborator (#22391)
add 86d0564ace0 Add accept-language header for MPL license (#22395)
add 5773178eabd BEAM-14525 Fix for Protobuf getter/setter method name discovery issue
add 5d482c31030 BEAM-14525 Added a proto message with the problematic properties to use a generated class as well for testing, and not only the current state
add 944056c9a7a Merge branch 'master' into BEAM-14525
add b5144a22f0a PR CR: updating issue links
add f2f239a44f4 Merge pull request #17775 from nbali/BEAM-14525
add 0581c49575e Bump terser from 5.9.0 to 5.14.2 in /sdks/python/apache_beam/runners/interactive/extensions/apache-beam-jupyterlab-sidepanel (#22365)
add 72127f93f45 Fixes #22156: Fix Spark3 runner to compile against Spark 3.2/3.3 and add version tests to verify compatibility going forward (#22157)
add 8d59cf7de71 Closes #22407: Separate sources for SparkStructuredStreamingRunner for Spark 2 & 3 to allow for easier improvements for Spark 3.
add 79d677fed70 Add deprecation warning for Spark 2 in SparkStructuredStreamingRunner
add b9f6af54d52 Merge pull request #22408 from mosche/22407-separate-spark-ssrunner-sources
add 23e10ebc5d7 Bump cloud.google.com/go/storage from 1.23.0 to 1.24.0 in /sdks (#22377)
add 35162273cf4 Bump cloud.google.com/go/bigquery from 1.35.0 to 1.36.0 in /sdks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4023e7336fb)
\
N -- N -- N refs/heads/dependabot/go_modules/sdks/cloud.google.com/go/bigquery-1.36.0 (35162273cf4)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 1 +
.../job_PreCommit_Java_Spark3_Versions.groovy | 25 +-
build.gradle.kts | 1 +
.../spark/structuredstreaming/Constants.java | 0
.../SparkStructuredStreamingPipelineOptions.java | 7 +
.../SparkStructuredStreamingPipelineResult.java | 39 +--
.../SparkStructuredStreamingRunner.java | 13 +-
.../SparkStructuredStreamingRunnerRegistrar.java | 0
.../aggregators/AggregatorsAccumulator.java | 0
.../aggregators/NamedAggregators.java | 0
.../aggregators/NamedAggregatorsAccumulator.java | 0
.../aggregators/package-info.java | 0
.../structuredstreaming/examples/WordCount.java | 0
.../metrics/AggregatorMetric.java | 74 ++++++
.../metrics/AggregatorMetricSource.java | 0
.../structuredstreaming/metrics/BeamMetricSet.java | 60 +++++
.../metrics/CompositeSource.java | 0
.../metrics/MetricsAccumulator.java | 0
.../MetricsContainerStepMapAccumulator.java | 0
.../metrics/SparkBeamMetric.java | 107 ++++++++
.../metrics/SparkBeamMetricSource.java | 0
.../metrics/SparkMetricsContainerStepMap.java | 0
.../metrics/WithMetricsSupport.java | 91 +++++++
.../structuredstreaming/metrics/package-info.java | 0
.../metrics/sink/CodahaleCsvSink.java | 86 +++++++
.../metrics/sink/CodahaleGraphiteSink.java | 89 +++++++
.../metrics/sink/package-info.java | 0
.../spark/structuredstreaming/package-info.java | 0
.../translation/AbstractTranslationContext.java | 25 +-
.../translation/PipelineTranslator.java | 0
.../translation/SparkSessionFactory.java | 71 ++++++
.../translation/SparkTransformOverrides.java | 0
.../translation/TransformTranslator.java | 0
.../translation/batch/AggregatorCombiner.java | 0
.../batch/CombinePerKeyTranslatorBatch.java | 0
.../CreatePCollectionViewTranslatorBatch.java | 0
.../translation/batch/DoFnFunction.java | 0
.../translation/batch/DoFnRunnerWithMetrics.java | 0
.../translation/batch/FlattenTranslatorBatch.java | 0
.../batch/GroupByKeyTranslatorBatch.java | 0
.../translation/batch/ImpulseTranslatorBatch.java | 0
.../translation/batch/ParDoTranslatorBatch.java | 0
.../translation/batch/PipelineTranslatorBatch.java | 0
.../translation/batch/ProcessContext.java | 0
.../batch/ReadSourceTranslatorBatch.java | 0
.../batch/ReshuffleTranslatorBatch.java | 0
.../batch/WindowAssignTranslatorBatch.java | 0
.../GroupAlsoByWindowViaOutputBufferFn.java | 0
.../batch/functions/NoOpStepContext.java | 0
.../batch/functions/SparkSideInputReader.java | 0
.../translation/batch/functions/package-info.java | 0
.../translation/batch/package-info.java | 0
.../translation/helpers/CoderHelpers.java | 0
.../translation/helpers/EncoderFactory.java | 43 ++--
.../translation/helpers/EncoderHelpers.java | 71 ++++++
.../translation/helpers/KVHelpers.java | 0
.../translation/helpers/MultiOutputCoder.java | 0
.../translation/helpers/RowHelpers.java | 0
.../translation/helpers/SchemaHelpers.java | 0
.../translation/helpers/SideInputBroadcast.java | 0
.../translation/helpers/WindowingHelpers.java | 0
.../translation/helpers/package-info.java | 0
.../translation/package-info.java | 0
.../streaming/PipelineTranslatorStreaming.java | 0
.../streaming/ReadSourceTranslatorStreaming.java | 0
.../translation/streaming/package-info.java | 0
.../translation/utils/CachedSideInputReader.java | 0
.../translation/utils/SideInputStorage.java | 0
.../translation/utils/package-info.java | 0
.../structuredstreaming/SparkSessionRule.java | 21 +-
...parkStructuredStreamingRunnerRegistrarTest.java | 0
.../StructuredStreamingPipelineStateTest.java | 0
.../aggregators/metrics/sink/InMemoryMetrics.java | 36 ++-
.../metrics/sink/InMemoryMetricsSinkRule.java | 0
.../metrics/sink/SparkMetricsSinkTest.java | 45 ++--
.../metrics/SparkBeamMetricTest.java} | 22 +-
.../translation/batch/CombineTest.java | 0
.../translation/batch/ComplexSourceTest.java | 0
.../translation/batch/FlattenTest.java | 0
.../translation/batch/GroupByKeyTest.java | 0
.../translation/batch/ParDoTest.java | 0
.../translation/batch/SimpleSourceTest.java | 0
.../translation/batch/WindowAssignTest.java | 0
.../translation/helpers/EncoderHelpersTest.java | 98 ++++++++
.../translation/streaming/SimpleSourceTest.java | 0
.../utils/SerializationDebugger.java | 0
.../structuredstreaming/utils/package-info.java | 0
runners/spark/3/build.gradle | 34 +++
.../spark/structuredstreaming/Constants.java | 0
.../SparkStructuredStreamingPipelineOptions.java | 7 +
.../SparkStructuredStreamingPipelineResult.java | 39 +--
.../SparkStructuredStreamingRunner.java | 9 +-
.../SparkStructuredStreamingRunnerRegistrar.java | 0
.../aggregators/AggregatorsAccumulator.java | 0
.../aggregators/NamedAggregators.java | 0
.../aggregators/NamedAggregatorsAccumulator.java | 0
.../aggregators/package-info.java | 0
.../structuredstreaming/examples/WordCount.java | 0
.../metrics/AggregatorMetric.java | 74 ++++++
.../metrics/AggregatorMetricSource.java | 0
.../structuredstreaming/metrics/BeamMetricSet.java | 60 +++++
.../metrics/CompositeSource.java | 0
.../metrics/MetricsAccumulator.java | 0
.../MetricsContainerStepMapAccumulator.java | 0
.../metrics/SparkBeamMetric.java | 107 ++++++++
.../metrics/SparkBeamMetricSource.java | 0
.../metrics/SparkMetricsContainerStepMap.java | 0
.../metrics/WithMetricsSupport.java | 91 +++++++
.../structuredstreaming/metrics/package-info.java | 0
.../metrics/sink/CodahaleCsvSink.java | 86 +++++++
.../metrics/sink/CodahaleGraphiteSink.java | 89 +++++++
.../metrics/sink/package-info.java | 0
.../spark/structuredstreaming/package-info.java | 0
.../translation/AbstractTranslationContext.java | 25 +-
.../translation/PipelineTranslator.java | 0
.../translation/SparkSessionFactory.java | 71 ++++++
.../translation/SparkTransformOverrides.java | 0
.../translation/TransformTranslator.java | 0
.../translation/batch/AggregatorCombiner.java | 0
.../batch/CombinePerKeyTranslatorBatch.java | 0
.../CreatePCollectionViewTranslatorBatch.java | 0
.../translation/batch/DatasetSourceBatch.java | 2 +-
.../translation/batch/DoFnFunction.java | 0
.../translation/batch/DoFnRunnerWithMetrics.java | 0
.../translation/batch/FlattenTranslatorBatch.java | 0
.../batch/GroupByKeyTranslatorBatch.java | 0
.../translation/batch/ImpulseTranslatorBatch.java | 0
.../translation/batch/ParDoTranslatorBatch.java | 0
.../translation/batch/PipelineTranslatorBatch.java | 0
.../translation/batch/ProcessContext.java | 0
.../batch/ReadSourceTranslatorBatch.java | 0
.../batch/ReshuffleTranslatorBatch.java | 0
.../batch/WindowAssignTranslatorBatch.java | 0
.../GroupAlsoByWindowViaOutputBufferFn.java | 0
.../batch/functions/NoOpStepContext.java | 0
.../batch/functions/SparkSideInputReader.java | 0
.../translation/batch/functions/package-info.java | 0
.../translation/batch/package-info.java | 0
.../translation/helpers/CoderHelpers.java | 0
.../translation/helpers/EncoderFactory.java | 59 +++--
.../translation/helpers/EncoderHelpers.java | 71 ++++++
.../translation/helpers/KVHelpers.java | 0
.../translation/helpers/MultiOutputCoder.java | 0
.../translation/helpers/RowHelpers.java | 0
.../translation/helpers/SchemaHelpers.java | 0
.../translation/helpers/SideInputBroadcast.java | 0
.../translation/helpers/WindowingHelpers.java | 0
.../translation/helpers/package-info.java | 0
.../translation/package-info.java | 0
.../streaming/PipelineTranslatorStreaming.java | 0
.../streaming/ReadSourceTranslatorStreaming.java | 0
.../translation/streaming/package-info.java | 0
.../translation/utils/CachedSideInputReader.java | 0
.../translation/utils/SideInputStorage.java | 0
.../translation/utils/package-info.java | 0
.../structuredstreaming/SparkSessionRule.java | 21 +-
...parkStructuredStreamingRunnerRegistrarTest.java | 0
.../StructuredStreamingPipelineStateTest.java | 0
.../aggregators/metrics/sink/InMemoryMetrics.java | 36 ++-
.../metrics/sink/InMemoryMetricsSinkRule.java | 0
.../metrics/sink/SparkMetricsSinkTest.java | 45 ++--
.../metrics/SparkBeamMetricTest.java} | 22 +-
.../translation/batch/CombineTest.java | 0
.../translation/batch/ComplexSourceTest.java | 0
.../translation/batch/FlattenTest.java | 0
.../translation/batch/GroupByKeyTest.java | 0
.../translation/batch/ParDoTest.java | 0
.../translation/batch/SimpleSourceTest.java | 0
.../translation/batch/WindowAssignTest.java | 0
.../translation/helpers/EncoderHelpersTest.java | 98 ++++++++
.../translation/streaming/SimpleSourceTest.java | 0
.../utils/SerializationDebugger.java | 0
.../structuredstreaming/utils/package-info.java | 0
runners/spark/spark_runner.gradle | 48 ++--
.../runners/spark/metrics/AggregatorMetric.java | 47 +++-
.../beam/runners/spark/metrics/BeamMetricSet.java | 59 +++++
.../runners/spark/metrics/SparkBeamMetric.java | 93 ++++---
.../runners/spark/metrics/WithMetricsSupport.java | 123 ++-------
.../beam/runners/spark/metrics/sink/CsvSink.java | 59 ++++-
.../runners/spark/metrics/sink/GraphiteSink.java | 66 ++++-
.../metrics/AggregatorMetric.java | 39 ---
.../metrics/SparkBeamMetric.java | 89 -------
.../metrics/WithMetricsSupport.java | 182 --------------
.../metrics/sink/CodahaleCsvSink.java | 36 ---
.../metrics/sink/CodahaleGraphiteSink.java | 34 ---
.../translation/helpers/EncoderHelpers.java | 274 ---------------------
.../aggregators/metrics/sink/InMemoryMetrics.java | 33 +--
.../metrics/sink/SparkMetricsSinkTest.java | 6 +-
.../runners/spark/metrics/SparkBeamMetricTest.java | 18 +-
.../translation/helpers/EncoderHelpersTest.java | 47 ----
.../spark/src/test/resources/metrics.properties | 68 -----
sdks/go.mod | 3 +-
sdks/go.sum | 8 +-
.../license_scripts/pull_licenses_java.py | 7 +-
.../extensions/protobuf/ProtoByteBuddyUtils.java | 51 +++-
.../protobuf/ProtoByteBuddyUtilsTest.java | 78 ++++++
.../proto/proto_byte_buddy_utils_messages.proto | 19 +-
sdks/java/io/sparkreceiver/build.gradle | 2 +
.../beam/sdk/io/sparkreceiver/HasOffset.java | 15 +-
.../apache-beam-jupyterlab-sidepanel/yarn.lock | 89 ++++---
.../typescript/src/apache_beam/coders/js_coders.ts | 6 +-
sdks/typescript/src/apache_beam/internal/urns.ts | 2 +
.../src/apache_beam/runners/direct_runner.ts | 39 ++-
.../src/apache_beam/transforms/combiners.ts | 4 +
.../apache_beam/transforms/group_and_combine.ts | 6 +-
.../src/apache_beam/transforms/internal.ts | 85 ++++++-
.../typescript/src/apache_beam/worker/operators.ts | 268 +++++++++++++++++++-
sdks/typescript/test/combine_test.ts | 17 +-
sdks/typescript/test/js_coders_test.ts | 1 +
settings.gradle.kts | 1 +
.../site/content/en/documentation/runners/spark.md | 6 +-
211 files changed, 2753 insertions(+), 1346 deletions(-)
copy runners/spark/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/InMemoryMetricsSinkRule.java => .test-infra/jenkins/job_PreCommit_Java_Spark3_Versions.groovy (63%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/Constants.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingPipelineOptions.java (85%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingPipelineResult.java (83%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunner.java (95%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunnerRegistrar.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/AggregatorsAccumulator.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/NamedAggregators.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/NamedAggregatorsAccumulator.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/package-info.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/examples/WordCount.java (100%)
create mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/AggregatorMetric.java
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/AggregatorMetricSource.java (100%)
create mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/BeamMetricSet.java
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/CompositeSource.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/MetricsAccumulator.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/MetricsContainerStepMapAccumulator.java (100%)
create mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetric.java
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetricSource.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkMetricsContainerStepMap.java (100%)
create mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/WithMetricsSupport.java
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/package-info.java (100%)
create mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/CodahaleCsvSink.java
create mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/CodahaleGraphiteSink.java
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/package-info.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/package-info.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/AbstractTranslationContext.java (88%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java (100%)
create mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/SparkSessionFactory.java
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/SparkTransformOverrides.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/TransformTranslator.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/AggregatorCombiner.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombinePerKeyTranslatorBatch.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CreatePCollectionViewTranslatorBatch.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DoFnFunction.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DoFnRunnerWithMetrics.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/FlattenTranslatorBatch.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyTranslatorBatch.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ImpulseTranslatorBatch.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ParDoTranslatorBatch.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/PipelineTranslatorBatch.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ProcessContext.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ReadSourceTranslatorBatch.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ReshuffleTranslatorBatch.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/WindowAssignTranslatorBatch.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/GroupAlsoByWindowViaOutputBufferFn.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/NoOpStepContext.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/SparkSideInputReader.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/package-info.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/package-info.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/CoderHelpers.java (100%)
create mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpers.java
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/KVHelpers.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/MultiOutputCoder.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/RowHelpers.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/SchemaHelpers.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/SideInputBroadcast.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/WindowingHelpers.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/package-info.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/package-info.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/PipelineTranslatorStreaming.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/ReadSourceTranslatorStreaming.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/package-info.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/CachedSideInputReader.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/SideInputStorage.java (100%)
copy runners/spark/{ => 2}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/package-info.java (100%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/SparkSessionRule.java (75%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunnerRegistrarTest.java (100%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/StructuredStreamingPipelineStateTest.java (100%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/InMemoryMetrics.java (75%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/InMemoryMetricsSinkRule.java (100%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/SparkMetricsSinkTest.java (65%)
copy runners/spark/{src/test/java/org/apache/beam/runners/spark/structuredstreaming/metrics/BeamMetricTest.java => 2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetricTest.java} (71%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombineTest.java (100%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ComplexSourceTest.java (100%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/FlattenTest.java (100%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyTest.java (100%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ParDoTest.java (100%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/SimpleSourceTest.java (100%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/WindowAssignTest.java (100%)
create mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpersTest.java
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/SimpleSourceTest.java (100%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/utils/SerializationDebugger.java (100%)
copy runners/spark/{ => 2}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/utils/package-info.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/Constants.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingPipelineOptions.java (85%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingPipelineResult.java (83%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunner.java (97%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunnerRegistrar.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/AggregatorsAccumulator.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/NamedAggregators.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/NamedAggregatorsAccumulator.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/package-info.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/examples/WordCount.java (100%)
create mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/AggregatorMetric.java
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/AggregatorMetricSource.java (100%)
create mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/BeamMetricSet.java
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/CompositeSource.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/MetricsAccumulator.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/MetricsContainerStepMapAccumulator.java (100%)
create mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetric.java
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetricSource.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkMetricsContainerStepMap.java (100%)
create mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/WithMetricsSupport.java
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/package-info.java (100%)
create mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/CodahaleCsvSink.java
create mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/CodahaleGraphiteSink.java
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/package-info.java (100%)
copy runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/package-info.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/AbstractTranslationContext.java (88%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java (100%)
create mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/SparkSessionFactory.java
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/SparkTransformOverrides.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/TransformTranslator.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/AggregatorCombiner.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombinePerKeyTranslatorBatch.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CreatePCollectionViewTranslatorBatch.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DoFnFunction.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DoFnRunnerWithMetrics.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/FlattenTranslatorBatch.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyTranslatorBatch.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ImpulseTranslatorBatch.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ParDoTranslatorBatch.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/PipelineTranslatorBatch.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ProcessContext.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ReadSourceTranslatorBatch.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ReshuffleTranslatorBatch.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/WindowAssignTranslatorBatch.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/GroupAlsoByWindowViaOutputBufferFn.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/NoOpStepContext.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/SparkSideInputReader.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/package-info.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/package-info.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/CoderHelpers.java (100%)
create mode 100644 runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpers.java
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/KVHelpers.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/MultiOutputCoder.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/RowHelpers.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/SchemaHelpers.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/SideInputBroadcast.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/WindowingHelpers.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/package-info.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/package-info.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/PipelineTranslatorStreaming.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/ReadSourceTranslatorStreaming.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/package-info.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/CachedSideInputReader.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/SideInputStorage.java (100%)
rename runners/spark/{ => 3}/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/package-info.java (100%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/SparkSessionRule.java (75%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunnerRegistrarTest.java (100%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/StructuredStreamingPipelineStateTest.java (100%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/InMemoryMetrics.java (75%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/InMemoryMetricsSinkRule.java (100%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/SparkMetricsSinkTest.java (65%)
rename runners/spark/{src/test/java/org/apache/beam/runners/spark/structuredstreaming/metrics/BeamMetricTest.java => 3/src/test/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetricTest.java} (71%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombineTest.java (100%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ComplexSourceTest.java (100%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/FlattenTest.java (100%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyTest.java (100%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ParDoTest.java (100%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/SimpleSourceTest.java (100%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/WindowAssignTest.java (100%)
create mode 100644 runners/spark/3/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpersTest.java
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/SimpleSourceTest.java (100%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/utils/SerializationDebugger.java (100%)
rename runners/spark/{ => 3}/src/test/java/org/apache/beam/runners/spark/structuredstreaming/utils/package-info.java (100%)
create mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/BeamMetricSet.java
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/AggregatorMetric.java
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetric.java
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/WithMetricsSupport.java
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/CodahaleCsvSink.java
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/CodahaleGraphiteSink.java
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpers.java
delete mode 100644 runners/spark/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpersTest.java
delete mode 100644 runners/spark/src/test/resources/metrics.properties
create mode 100644 sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoByteBuddyUtilsTest.java
copy runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/package-info.java => sdks/java/extensions/protobuf/src/test/proto/proto_byte_buddy_utils_messages.proto (68%)
rename runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/package-info.java => sdks/java/io/sparkreceiver/src/main/java/org/apache/beam/sdk/io/sparkreceiver/HasOffset.java (64%)