You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ke...@apache.org on 2018/01/26 02:50:15 UTC
[beam] branch samza-runner updated (a539811 -> 0f554fd)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch samza-runner
in repository https://gitbox.apache.org/repos/asf/beam.git.
from a539811 Merge pull request #4451 from robertwb/fn-api-pickle-side
add 6375069 [BEAM-2762] Python code coverage report in Postcommit
add d41da38 fixup! Exclude auto-generated files in portability/api
add 3ec529b fixup! Clean up configuration and use better way to exclude cover in mvn build
add a2b409c fixup! Add more comments
add 3f53993 fixup! Improve comments
add 509ad27 Merge pull request #3879 from markflyhigh/coverage-report
add c5c0cce [BEAM-2963] Remove layer of indirection in output name mapping in Dataflow simplifying what needs to be passed for all portable pipelines to the Java SDK harness. (#4460)
add 798da37 Add a BundleProcessor to SdkHarnessClient
add 55844fa Merge pull request #4266
add 4719468 Curry CombineFn arguments into runner API protos.
add 94d5c0b Merge pull request #4437 from robertwb/runner-api-combine-fn
add 3be0467 Disable combiner lifting optimization in DataflowRunner for fnapi
add a69f6b6 Disable combiner lifting when only the streaming flag is set.
add 7a4e98f Merge pull request #4455 from aaltay/comb
add 10a6e92 Added an example pipeline that uses stateful processing to output team score every time it passes a new multiple of a threshold.
add ae1f547 Merge pull request #4272
add b24800d Register Environments in SdkComponents
add a16abdf Merge pull request #4464
add 1057d9a Update BUILD.md
add 02e9a96 Merge pull request #4399 from pshevtsov/patch-1
add 23ba6ba NotImplementedErrror --> NotImplementedError
add 584292e Merge pull request #4359 from cclauss/patch-1
add ab85e0c Fix code comment to match code
add 19663de Merge pull request #4278 from itamaro/patch-1
add 9641567 Manually specify direct runner for global-state modifying tests.
add ade4fbd FakeSource requires direct runner.
add 194c393 Explicitly use DirectRunner in DirectRunner tests.
add 4f92495 Document DirectRunnerOnly tests.
add 3920cd7 Merge pull request #4454 [BEAM-3490] Explicitly use direct runner in non-portable tests.
add 9b10739 Remove legacy windowfn translation.
add d682042 Merge pull request #4212 Remove legacy windowfn translation.
add e8a725e Pass PipelineOptions to FileSystem constructor.
add 6663e60 Merge pull request #4465 from udim/filesystem-options
add f2503ba [BEAM-3008] Adds parameters templatization for Bigtable (#4357)
add 6affcb5 Migrate Flink ValidatesRunner to Gradle
add 57e8525 Increment the Dataflow runner major version to 7.
add 0e50e7d Update copyright date in NOTICE
add 62c266a Merge pull request #4477 from yew1eb/update_copyright_date
add 2e4b1c2 Move BeamFnDataInboundObserver to java-fn-execution
add 7be579e Add an implementation of the Fn API Data Service
add f83ac0b Merge pull request #4453
add e70b068 [BEAM-3126] Adding a new Flatten test to Python SDK. (#4463)
add 51ca1ee Code compiles after auto-transition to lambda
add 3871bbe google-java-format
add c824716 Removes unnecessary explicit type arguments
add 1242a83 google-java-format
add 10b5ad0 checkstyle fixups
add 3d06c89 More removal of explicit type arguments
add b0a2b0f google-java-format
add 1ca3a4d Manually fixed a couple cases of bad comment formatting
add aedb4c8 Manual fixup of some call sites where lambdas mess up coder inference
add 0d87135 A couple of final example fixups
add 0cbcf4a Merge pull request #4470 from jkff/java8
add cb293b9 Update printed gcloud cancel commands to include the job's region.
add 12e30a3 Merge pull request #4473
add 6389f38 [BEAM-3412] Upgrade Cloud Bigtable to 1.0.0 The 1.0.0 Cloud Bigtable client upgrade included a non-backwards compatible change in protobuf generation. Specifically, addAllXXXX() methods no longer work correctly. `BigtableServiceImpl` used `addAllMutations()`, which caused run-time breakages. To get around that, add each Mutation independently to the protobuf object. This PR also adds unit tests for BigtableServiceImpl so that read/write interactions do not cause runti [...]
add 1559e37 Merge remote-tracking branch 'upstream/master' into upgrade_cbt_100
add 0372db0 BigtableServiceImplTest now uses a List of ranges. The PR did not previously compile, since I had a pretty old version of the code in my development environment. There was a non-backwards compatible change that I had to fix.
add 42868b5 Updating Cloud Bigtable dependnecy to 1.0.0 in build.gradle.
add 1095ac5 Using 1.0.0-pre3 for bigtable-proto. This should allow Cloud Bigtable's client to upgrade to 1.0.0 while keeping backwards compatibility with Beam's protobuf version.
add c0da25c Merge branch 'master' into upgrade_cbt_100
add bd8d075 Updating build.gradle with a bigtable_proto_version of 1.0.0-pre3
add 98d32ad Fixing a bad merge in BigtableServiceImpl.
add e39964c Merge pull request #4462 from sduskis/upgrade_cbt_100
add f7dd050 Upgrade Jenkins jobs to use Maven 3.5.2
add 3474505 [BEAM-2728] Add Count-Min Sketch in sketching extension
add eabc640 Small fix in SketchCoder
add cc9182a Make Sketch AutoValue + Javadoc update
add 7f99628 Optimize coder memory use
add 5b23599 Add CONTRIBUTING.md
add 1e3f1c8 Merge pull request #4475: Add CONTRIBUTING.md
add 93f4bd4 Initial post-release snapshot test
add 1c30f40 Make the snapshot and url parameters to the job to allow installing a different version
add ec2b0e1 Make the snapshot and url parameters to the job to allow installing a different version
add 3c6899c Rename TestHelper to TestScripts, it() to intent()
add ad82ac1 [BEAM-3060] Support for Perfkit execution of file-based-io-tests on HDFS cluster. (#4401)
add 8af534f [BEAM-3428] Merge Java8 examples in "main" Java examples
add e2f5061 [BEAM-3428] Replace MinimalWordCount with Java8 one
add f2ed860 Merge pull request #4479 from: [BEAM-3428] Merge Java8 examples in "main" Java examples
add fcf49e9 [BEAM-3534] Add a spark validates runner test for metrics sink in streaming mode
add 44b164d Merge pull request #4361: [BEAM-3534] Add a spark validates runner test for metrics sink in streaming mode
add 3abaaac [BEAM-3432] Remove hadoop/jdk1.8-tests module
add a905019 Merge pull request #4467 from iemejia/BEAM-3432-hifi-java8
add 9e826f1 [BEAM-3160] Fix issue where we would choose which coder to use arbitrarily when it is over specified with multiple coders.
add c07a43e Merge pull request #4433: [BEAM-3160] Prevent issue where we would choose which coder to use arbitrarily when it is over specified with multiple coders
add 21af2d3 BEAM-3533 - Replace hard-coded UTF-8 Strings
add e28c4da Merge pull request #4486 from coheigea/BEAM-3533
add dafe47d [BEAM-3466] Remove JDK 7 references in Jenkins
add 5ee938e Merge pull request #4489: [BEAM-3466] Remove JDK 7 references in Jenkins
add 7061301 fix serialization error in BigQueryIO's DynamicDestinations
add aaabb64 correct side input check in BigQueryIO DynamicDestination for pull
add 0f554fd Merge pull request #4417 from shashank64/master
No new revisions were added by this update.
Summary of changes:
.test-infra/jenkins/common_job_properties.groovy | 2 +-
...job_beam_PostCommit_Java_JDKVersionsTest.groovy | 4 +-
...eam_PostCommit_Java_MavenInstall_Windows.groovy | 2 +-
...=> job_beam_PostRelease_NightlySnapshot.groovy} | 47 +-
.../SmallITCluster/pkb-config.yml} | 24 +-
sdks/java/io/kafka/README.md => CONTRIBUTING.md | 20 +-
NOTICE | 2 +-
build.gradle | 5 +-
examples/java/pom.xml | 5 +
.../org/apache/beam/examples/MinimalWordCount.java | 88 +-
.../apache/beam/examples/WindowedWordCount.java | 4 +-
.../java/org/apache/beam/examples/WordCount.java | 3 +-
.../beam/examples/complete/AutoComplete.java | 78 +-
.../org/apache/beam/examples/complete/TfIdf.java | 50 +-
.../examples/complete/TopWikipediaSessions.java | 17 +-
.../beam/examples/complete/TrafficMaxLaneFlow.java | 14 +-
.../beam/examples/complete/TrafficRoutes.java | 17 +-
.../beam/examples/complete/game/GameStats.java | 172 ++--
.../examples/complete/game/HourlyTeamScore.java | 80 +-
.../beam/examples/complete/game/LeaderBoard.java | 50 +-
.../examples/complete/game/StatefulTeamScore.java | 228 +++++
.../beam/examples/complete/game/UserScore.java | 18 +-
.../examples/complete/game/injector/Injector.java | 72 +-
.../complete/game/injector/InjectorUtils.java | 0
.../game/injector/RetryHttpInitializerWrapper.java | 42 +-
.../complete/game/utils/GameConstants.java | 23 +-
.../complete/game/utils/WriteToBigQuery.java | 0
.../examples/complete/game/utils/WriteToText.java | 2 +-
.../game/utils/WriteWindowedToBigQuery.java | 0
.../beam/examples/cookbook/BigQueryTornadoes.java | 3 +-
.../examples/cookbook/CombinePerKeyExamples.java | 4 +-
.../beam/examples/cookbook/DistinctExample.java | 4 +-
.../beam/examples/cookbook/FilterExamples.java | 3 +-
.../beam/examples/cookbook/JoinExamples.java | 12 +-
.../beam/examples/cookbook/MaxPerKeyExamples.java | 3 +-
.../beam/examples/cookbook/TriggerExample.java | 4 +-
.../beam/examples/website_snippets/Snippets.java | 11 +-
.../beam/examples/MinimalWordCountTest.java} | 57 +-
.../org/apache/beam/examples/WordCountTest.java | 3 +-
.../beam/examples/complete/AutoCompleteTest.java | 17 +-
.../apache/beam/examples/complete/TfIdfTest.java | 4 +-
.../beam/examples/complete/game/GameStatsTest.java | 0
.../complete/game/HourlyTeamScoreTest.java | 0
.../examples/complete/game/LeaderBoardTest.java | 16 +-
.../complete/game/StatefulTeamScoreTest.java | 208 +++++
.../beam/examples/complete/game/UserScoreTest.java | 0
.../examples/cookbook/DistinctExampleTest.java | 6 +-
.../beam/examples/cookbook/TriggerExampleTest.java | 6 +-
.../examples/website_snippets/SnippetsTest.java | 3 +-
examples/java8/build.gradle | 50 --
examples/java8/pom.xml | 296 -------
.../beam/examples/MinimalWordCountJava8.java | 72 --
examples/pom.xml | 23 -
pom.xml | 11 +-
release/TestScripts.groovy | 140 +++
release/quickstart-java-direct.groovy | 64 ++
.../org/apache/beam/runners/apex/ApexRunner.java | 22 +-
.../beam/runners/apex/ApexRunnerRegistrar.java | 5 +-
.../apache/beam/runners/apex/ApexYarnLauncher.java | 7 +-
.../runners/apex/translation/ParDoTranslator.java | 2 +-
.../operators/ApexGroupByKeyOperator.java | 43 +-
.../translation/operators/ApexParDoOperator.java | 10 +-
.../operators/ApexProcessFnOperator.java | 6 +-
.../operators/ApexReadUnboundedInputOperator.java | 2 +-
.../apex/translation/utils/ApexStateInternals.java | 2 +-
.../apache/beam/runners/apex/ApexRunnerTest.java | 2 +-
.../beam/runners/apex/ApexYarnLauncherTest.java | 8 +-
.../beam/runners/apex/examples/WordCountTest.java | 14 +-
.../FlattenPCollectionTranslatorTest.java | 9 +-
.../apex/translation/GroupByKeyTranslatorTest.java | 4 +-
.../apex/translation/ParDoTranslatorTest.java | 52 +-
.../apex/translation/SideInputTranslationTest.java | 10 +-
.../operators/ApexTimerInternalsTest.java | 7 +-
.../core/construction/ArtifactServiceStager.java | 3 +-
.../core/construction/CombineTranslation.java | 15 +-
.../CreatePCollectionViewTranslation.java | 5 +-
.../construction/DeduplicatedFlattenFactory.java | 6 +-
.../construction/EmptyFlattenAsCreateFactory.java | 2 +-
...dTranslatorRegistrar.java => Environments.java} | 20 +-
.../core/construction/PTransformMatchers.java | 45 +-
.../core/construction/PTransformTranslation.java | 3 +-
.../core/construction/ParDoTranslation.java | 18 +-
.../core/construction/PipelineTranslation.java | 3 +-
.../runners/core/construction/PrimitiveCreate.java | 2 +-
.../runners/core/construction/ReadTranslation.java | 35 +-
.../core/construction/RehydratedComponents.java | 8 +
.../runners/core/construction/SdkComponents.java | 22 +-
.../runners/core/construction/SplittableParDo.java | 22 +-
.../UnboundedReadFromBoundedSource.java | 16 +-
.../runners/core/construction/UnconsumedReads.java | 2 +-
.../core/construction/WindowIntoTranslation.java | 4 +-
.../construction/WindowingStrategyTranslation.java | 4 +-
.../core/construction/WriteFilesTranslation.java | 8 +-
.../construction/ArtifactServiceStagerTest.java | 2 +-
.../core/construction/CoderTranslationTest.java | 2 +-
.../core/construction/CombineTranslationTest.java | 2 +-
.../CreatePCollectionViewTranslationTest.java | 12 +-
.../DeduplicatedFlattenFactoryTest.java | 8 +-
.../EmptyFlattenAsCreateFactoryTest.java | 31 +-
.../construction/ForwardingPTransformTest.java | 4 +-
.../construction/PCollectionTranslationTest.java | 22 +-
.../core/construction/PTransformMatchersTest.java | 146 ++--
.../construction/PTransformReplacementsTest.java | 12 +-
.../construction/PTransformTranslationTest.java | 9 +-
.../core/construction/ParDoTranslationTest.java | 39 +-
.../core/construction/PipelineTranslationTest.java | 9 +-
.../core/construction/ReadTranslationTest.java | 10 +-
.../construction/RehydratedComponentsTest.java | 14 +
.../core/construction/ReplacementOutputsTest.java | 9 +-
.../core/construction/SdkComponentsTest.java | 36 +-
.../SingleInputOutputOverrideFactoryTest.java | 2 +-
.../construction/TestStreamTranslationTest.java | 8 +-
.../core/construction/TransformInputsTest.java | 29 +-
.../UnboundedReadFromBoundedSourceTest.java | 18 +-
.../core/construction/UnconsumedReadsTest.java | 9 +-
.../UnsupportedOverrideFactoryTest.java | 4 +-
.../WindowingStrategyTranslationTest.java | 2 +-
.../construction/metrics/MetricFilteringTest.java | 2 +-
.../runners/core/GroupByKeyViaGroupByKeyOnly.java | 23 +-
.../beam/runners/core/InMemoryStateInternals.java | 4 +-
.../apache/beam/runners/core/KeyedWorkItems.java | 2 +-
.../runners/core/LateDataDroppingDoFnRunner.java | 35 +-
.../apache/beam/runners/core/LateDataUtils.java | 63 +-
.../beam/runners/core/MergingActiveWindowSet.java | 4 +-
.../beam/runners/core/NullSideInputReader.java | 2 +-
...TimeBoundedSplittableProcessElementInvoker.java | 7 +-
.../beam/runners/core/ReduceFnContextFactory.java | 15 +-
.../apache/beam/runners/core/ReduceFnRunner.java | 54 +-
.../core/SplittableParDoViaKeyedWorkItems.java | 2 +-
.../apache/beam/runners/core/SystemReduceFn.java | 21 +-
.../apache/beam/runners/core/WatermarkHold.java | 2 +-
.../beam/runners/core/metrics/MetricUpdates.java | 7 +-
.../runners/core/metrics/MetricsContainerImpl.java | 24 +-
.../core/metrics/MetricsContainerStepMap.java | 159 ++--
.../core/triggers/AfterAllStateMachine.java | 2 +-
.../AfterDelayFromFirstElementStateMachine.java | 12 +-
.../core/triggers/AfterEachStateMachine.java | 2 +-
.../core/triggers/AfterFirstStateMachine.java | 2 +-
...fterSynchronizedProcessingTimeStateMachine.java | 3 +-
.../core/triggers/AfterWatermarkStateMachine.java | 4 +-
.../TriggerStateMachineContextFactory.java | 11 +-
.../beam/runners/core/KeyedWorkItemCoderTest.java | 5 +-
.../beam/runners/core/LateDataUtilsTest.java | 3 +-
.../beam/runners/core/ReduceFnRunnerTest.java | 48 +-
.../apache/beam/runners/core/ReduceFnTester.java | 70 +-
.../beam/runners/core/SideInputHandlerTest.java | 49 +-
.../beam/runners/core/SimpleDoFnRunnerTest.java | 24 +-
.../SimplePushbackSideInputDoFnRunnerTest.java | 21 +-
.../runners/core/SplittableParDoProcessFnTest.java | 22 +-
.../beam/runners/core/StatefulDoFnRunnerTest.java | 6 +-
.../apache/beam/runners/core/WindowMatchers.java | 20 +-
.../core/metrics/MetricsContainerStepMapTest.java | 2 +-
.../beam/runners/core/metrics/MetricsMapTest.java | 8 +-
.../core/triggers/AfterFirstStateMachineTest.java | 2 +-
.../triggers/AfterWatermarkStateMachineTest.java | 4 +-
.../core/triggers/FinishedTriggersSetTest.java | 9 +-
.../core/triggers/RepeatedlyStateMachineTest.java | 7 +-
.../core/triggers/StubTriggerStateMachine.java | 2 +-
.../core/triggers/TriggerStateMachineTest.java | 10 +-
.../core/triggers/TriggerStateMachineTester.java | 62 +-
...DirectGBKIntoKeyedWorkItemsOverrideFactory.java | 2 +-
.../beam/runners/direct/DirectGroupByKey.java | 4 +-
.../apache/beam/runners/direct/DirectMetrics.java | 45 +-
.../beam/runners/direct/DirectRegistrar.java | 6 +-
.../beam/runners/direct/EvaluationContext.java | 8 +-
.../direct/ExecutorServiceParallelExecutor.java | 13 +-
.../direct/GroupAlsoByWindowEvaluatorFactory.java | 52 +-
.../direct/ImmutabilityEnforcementFactory.java | 2 +-
.../runners/direct/KeyedPValueTrackingVisitor.java | 2 +-
.../beam/runners/direct/MultiStepCombine.java | 2 +-
.../apache/beam/runners/direct/ParDoEvaluator.java | 44 +-
.../beam/runners/direct/ParDoEvaluatorFactory.java | 2 +-
.../runners/direct/ParDoMultiOverrideFactory.java | 8 +-
.../beam/runners/direct/QuiescenceDriver.java | 12 +-
.../beam/runners/direct/ReadEvaluatorFactory.java | 4 +-
.../beam/runners/direct/SideInputContainer.java | 18 +-
.../SplittableProcessElementsEvaluatorFactory.java | 78 +-
.../direct/StatefulParDoEvaluatorFactory.java | 31 +-
.../runners/direct/TestStreamEvaluatorFactory.java | 2 +-
.../direct/UnboundedReadEvaluatorFactory.java | 17 +-
.../beam/runners/direct/ViewOverrideFactory.java | 13 +-
.../beam/runners/direct/WatermarkManager.java | 5 +-
.../runners/direct/WindowEvaluatorFactory.java | 2 +-
.../runners/direct/WriteWithShardingFactory.java | 8 +-
.../direct/BoundedReadEvaluatorFactoryTest.java | 19 +-
.../runners/direct/CloningBundleFactoryTest.java | 9 +-
.../beam/runners/direct/CommittedResultTest.java | 20 +-
.../runners/direct/DirectGraphVisitorTest.java | 29 +-
.../DirectGroupByKeyOverrideFactoryTest.java | 3 +-
.../beam/runners/direct/DirectMetricsTest.java | 88 +-
.../beam/runners/direct/DirectRunnerTest.java | 107 ++-
.../direct/DirectTransformExecutorTest.java | 31 +-
...cycleManagerRemovingTransformEvaluatorTest.java | 6 +-
.../runners/direct/DoFnLifecycleManagerTest.java | 2 +-
.../beam/runners/direct/EvaluationContextTest.java | 62 +-
.../direct/FlattenEvaluatorFactoryTest.java | 12 +-
.../direct/GroupByKeyEvaluatorFactoryTest.java | 2 +-
.../direct/GroupByKeyOnlyEvaluatorFactoryTest.java | 2 +-
.../ImmutabilityCheckingBundleFactoryTest.java | 2 +-
.../direct/ImmutabilityEnforcementFactoryTest.java | 8 +-
.../direct/ImmutableListBundleFactoryTest.java | 10 +-
.../direct/KeyedPValueTrackingVisitorTest.java | 23 +-
.../beam/runners/direct/MultiStepCombineTest.java | 8 +-
.../beam/runners/direct/ParDoEvaluatorTest.java | 14 +-
.../runners/direct/SideInputContainerTest.java | 117 +--
.../direct/StatefulParDoEvaluatorFactoryTest.java | 32 +-
.../runners/direct/StepTransformResultTest.java | 3 +-
.../beam/runners/direct/StructuralKeyTest.java | 4 +-
.../direct/TestStreamEvaluatorFactoryTest.java | 7 +-
.../direct/UnboundedReadEvaluatorFactoryTest.java | 30 +-
.../runners/direct/ViewEvaluatorFactoryTest.java | 15 +-
.../runners/direct/ViewOverrideFactoryTest.java | 25 +-
.../beam/runners/direct/WatermarkManagerTest.java | 325 ++++---
.../runners/direct/WindowEvaluatorFactoryTest.java | 17 +-
.../direct/WriteWithShardingFactoryTest.java | 5 +-
runners/flink/build.gradle | 72 +-
.../runners/flink/CreateStreamingFlinkView.java | 7 +-
.../flink/FlinkBatchTransformTranslators.java | 49 +-
.../beam/runners/flink/FlinkRunnerRegistrar.java | 7 +-
.../beam/runners/flink/FlinkRunnerResult.java | 7 +-
.../flink/FlinkStreamingTransformTranslators.java | 171 ++--
.../FlinkMergingNonShuffleReduceFunction.java | 2 +-
.../functions/FlinkPartialReduceFunction.java | 3 +-
.../translation/functions/FlinkReduceFunction.java | 2 +-
.../functions/FlinkSideInputReader.java | 7 +-
.../functions/SideInputInitializer.java | 20 +-
.../functions/SortingFlinkCombineRunner.java | 16 +-
.../wrappers/streaming/DoFnOperator.java | 28 +-
.../wrappers/streaming/SplittableDoFnOperator.java | 35 +-
.../wrappers/streaming/WindowDoFnOperator.java | 36 +-
.../state/FlinkBroadcastStateInternals.java | 2 +-
.../state/FlinkKeyGroupStateInternals.java | 6 +-
.../streaming/state/FlinkSplitStateInternals.java | 2 +-
.../streaming/state/FlinkStateInternals.java | 39 +-
.../beam/runners/flink/PipelineOptionsTest.java | 52 +-
.../flink/streaming/DedupingOperatorTest.java | 46 +-
.../runners/flink/streaming/DoFnOperatorTest.java | 292 +++----
.../flink/streaming/GroupByNullKeyTest.java | 76 +-
.../streaming/TopWikipediaSessionsITCase.java | 122 ++-
.../runners/gearpump/GearpumpPipelineResult.java | 7 +-
.../runners/gearpump/GearpumpRunnerRegistrar.java | 7 +-
.../translators/CreateStreamingGearpumpView.java | 10 +-
.../translators/FlattenPCollectionsTranslator.java | 5 +-
.../gearpump/translators/GroupByKeyTranslator.java | 23 +-
runners/google-cloud-dataflow-java/build.gradle | 4 +-
runners/google-cloud-dataflow-java/pom.xml | 4 +-
.../dataflow/BatchStatefulParDoOverrides.java | 22 +-
.../beam/runners/dataflow/BatchViewOverrides.java | 173 ++--
.../beam/runners/dataflow/DataflowMetrics.java | 2 +-
.../beam/runners/dataflow/DataflowPipelineJob.java | 85 +-
.../dataflow/DataflowPipelineRegistrar.java | 6 +-
.../dataflow/DataflowPipelineTranslator.java | 65 +-
.../beam/runners/dataflow/DataflowRunner.java | 41 +-
.../dataflow/PrimitiveParDoSingleFactory.java | 7 +
.../beam/runners/dataflow/ReadTranslator.java | 2 +-
.../runners/dataflow/ReshuffleOverrideFactory.java | 30 +-
.../runners/dataflow/StreamingViewOverrides.java | 2 +-
.../beam/runners/dataflow/TransformTranslator.java | 17 +-
.../beam/runners/dataflow/internal/IsmFormat.java | 11 +-
.../dataflow/util/CloudObjectTranslators.java | 22 +-
...DefaultCoderCloudObjectTranslatorRegistrar.java | 2 +-
.../beam/runners/dataflow/util/MonitoringUtil.java | 12 +-
.../beam/runners/dataflow/util/PackageUtil.java | 58 +-
.../apache/beam/runners/dataflow/util/Structs.java | 10 +-
.../dataflow/BatchStatefulParDoOverridesTest.java | 13 +-
.../runners/dataflow/BatchViewOverridesTest.java | 12 +-
.../beam/runners/dataflow/DataflowMetricsTest.java | 2 +-
.../runners/dataflow/DataflowPipelineJobTest.java | 50 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 127 +--
.../beam/runners/dataflow/DataflowRunnerTest.java | 147 ++--
.../dataflow/PrimitiveParDoSingleFactoryTest.java | 2 +-
.../runners/dataflow/TestDataflowRunnerTest.java | 50 +-
.../transforms/DataflowGroupByKeyTest.java | 14 +-
.../dataflow/transforms/DataflowViewTest.java | 49 +-
.../runners/dataflow/util/CloudObjectsTest.java | 8 +-
.../runners/dataflow/util/MonitoringUtilTest.java | 12 +-
.../runners/dataflow/util/PackageUtilTest.java | 13 +-
.../beam/runners/dataflow/util/StructsTest.java | 11 +-
.../fnexecution/control/SdkHarnessClient.java | 190 ++--
.../runners/fnexecution/data/FnDataService.java | 5 +-
.../runners/fnexecution/data/GrpcDataService.java | 161 ++++
.../runners/fnexecution/ServerFactoryTest.java | 33 +-
.../fnexecution/control/SdkHarnessClientTest.java | 32 +-
.../fnexecution/data/GrpcDataServiceTest.java | 185 ++++
.../logging/GrpcLoggingServiceTest.java | 112 ++-
.../LocalFileSystemArtifactStagerServiceTest.java | 2 +-
.../beam/runners/local/StructuralKeyTest.java | 5 +-
.../reference/job/ReferenceRunnerJobService.java | 7 +-
.../job/ReferenceRunnerJobServiceTest.java | 12 +-
.../runners/spark/SparkNativePipelineVisitor.java | 13 +-
.../org/apache/beam/runners/spark/SparkRunner.java | 22 +-
.../beam/runners/spark/SparkRunnerRegistrar.java | 5 +-
.../apache/beam/runners/spark/TestSparkRunner.java | 2 +-
.../spark/aggregators/AggregatorsAccumulator.java | 6 +-
.../spark/aggregators/NamedAggregators.java | 12 +-
.../beam/runners/spark/coders/CoderHelpers.java | 60 +-
.../beam/runners/spark/examples/WordCount.java | 3 +-
.../apache/beam/runners/spark/io/CreateStream.java | 3 +-
.../beam/runners/spark/io/SourceDStream.java | 4 +-
.../apache/beam/runners/spark/io/SourceRDD.java | 5 +-
.../runners/spark/io/SparkUnboundedSource.java | 18 +-
.../runners/spark/metrics/MetricsAccumulator.java | 6 +-
.../runners/spark/metrics/WithMetricsSupport.java | 95 +-
.../SparkGroupAlsoByWindowViaWindowSet.java | 93 +-
.../spark/stateful/SparkTimerInternals.java | 3 +-
.../runners/spark/translation/BoundedDataset.java | 16 +-
.../spark/translation/EvaluationContext.java | 4 +-
.../spark/translation/GroupCombineFunctions.java | 145 ++--
.../spark/translation/MultiDoFnFunction.java | 20 +-
.../spark/translation/SparkAbstractCombineFn.java | 15 +-
.../spark/translation/SparkProcessContext.java | 8 +-
.../spark/translation/TransformTranslator.java | 274 +++---
.../spark/translation/TranslationUtils.java | 178 ++--
.../spark/translation/WindowingHelpers.java | 29 +-
.../streaming/StreamingTransformTranslator.java | 160 ++--
.../translation/streaming/UnboundedDataset.java | 11 +-
.../streaming/WatermarkSyncedDStream.java | 3 +-
.../runners/spark/util/GlobalWatermarkHolder.java | 2 +-
.../runners/spark/util/SparkSideInputReader.java | 27 +-
.../org/apache/beam/runners/spark/CacheTest.java | 4 +-
.../runners/spark/SparkRunnerDebuggerTest.java | 14 +-
.../spark/TestSparkPipelineOptionsRegistrar.java | 2 +-
.../metrics/sink/SparkMetricsSinkTest.java | 65 +-
.../beam/runners/spark/io/NumShardsTest.java | 12 +-
.../translation/streaming/CreateStreamTest.java | 97 +--
.../ResumeFromCheckpointStreamingTest.java | 62 +-
.../streaming/TrackStreamingSourcesTest.java | 3 +-
sdks/go/BUILD.md | 2 +-
.../src/main/resources/beam/findbugs-filter.xml | 4 +-
.../main/java/org/apache/beam/sdk/Pipeline.java | 3 +-
.../java/org/apache/beam/sdk/coders/AvroCoder.java | 8 +-
.../sdk/coders/CannotProvideCoderException.java | 10 +-
.../org/apache/beam/sdk/coders/CoderRegistry.java | 51 +-
.../org/apache/beam/sdk/coders/DefaultCoder.java | 2 +-
.../org/apache/beam/sdk/coders/DelegateCoder.java | 2 +-
.../beam/sdk/coders/StringDelegateCoder.java | 29 +-
.../main/java/org/apache/beam/sdk/io/AvroIO.java | 22 +-
.../main/java/org/apache/beam/sdk/io/AvroSink.java | 4 +-
.../java/org/apache/beam/sdk/io/AvroSource.java | 12 +-
.../sdk/io/BoundedReadFromUnboundedSource.java | 21 +-
.../org/apache/beam/sdk/io/CompressedSource.java | 3 +-
.../beam/sdk/io/DynamicAvroDestinations.java | 2 +-
.../java/org/apache/beam/sdk/io/FileBasedSink.java | 4 +-
.../main/java/org/apache/beam/sdk/io/FileIO.java | 122 +--
.../java/org/apache/beam/sdk/io/FileSystems.java | 103 +--
.../org/apache/beam/sdk/io/LocalFileSystem.java | 17 +-
.../beam/sdk/io/LocalFileSystemRegistrar.java | 2 +-
.../org/apache/beam/sdk/io/LocalResources.java | 8 +-
.../src/main/java/org/apache/beam/sdk/io/Read.java | 4 +-
.../beam/sdk/io/ReadAllViaFileBasedSource.java | 4 +-
.../java/org/apache/beam/sdk/io/TFRecordIO.java | 2 +-
.../main/java/org/apache/beam/sdk/io/TextIO.java | 27 +-
.../java/org/apache/beam/sdk/io/WriteFiles.java | 20 +-
.../org/apache/beam/sdk/io/WriteFilesResult.java | 4 +-
.../beam/sdk/metrics/MetricsEnvironment.java | 3 +-
.../beam/sdk/options/PipelineOptionsFactory.java | 123 +--
.../beam/sdk/options/ProxyInvocationHandler.java | 23 +-
.../org/apache/beam/sdk/options/ValueProvider.java | 2 +-
.../apache/beam/sdk/runners/AppliedPTransform.java | 3 +-
.../beam/sdk/runners/TransformHierarchy.java | 8 +-
.../java/org/apache/beam/sdk/state/StateSpecs.java | 12 +-
.../org/apache/beam/sdk/testing/Annotations.java | 34 +-
.../apache/beam/sdk/testing/CoderProperties.java | 11 +-
.../apache/beam/sdk/testing/GatherAllPanes.java | 6 +-
.../java/org/apache/beam/sdk/testing/PAssert.java | 103 +--
.../beam/sdk/testing/SerializableMatchers.java | 400 ++-------
.../apache/beam/sdk/testing/SourceTestUtils.java | 44 +-
.../org/apache/beam/sdk/testing/TestPipeline.java | 9 +-
.../org/apache/beam/sdk/testing/TestStream.java | 14 +-
.../apache/beam/sdk/testing/WindowFnTestUtils.java | 8 +-
.../beam/sdk/transforms/ApproximateQuantiles.java | 27 +-
.../beam/sdk/transforms/ApproximateUnique.java | 3 +-
.../org/apache/beam/sdk/transforms/Combine.java | 43 +-
.../org/apache/beam/sdk/transforms/Contextful.java | 9 +-
.../java/org/apache/beam/sdk/transforms/Count.java | 25 +-
.../org/apache/beam/sdk/transforms/Create.java | 23 +-
.../org/apache/beam/sdk/transforms/Distinct.java | 41 +-
.../org/apache/beam/sdk/transforms/DoFnTester.java | 42 +-
.../org/apache/beam/sdk/transforms/Filter.java | 41 +-
.../org/apache/beam/sdk/transforms/Latest.java | 51 +-
.../java/org/apache/beam/sdk/transforms/Max.java | 22 +-
.../java/org/apache/beam/sdk/transforms/Mean.java | 4 +-
.../java/org/apache/beam/sdk/transforms/Min.java | 22 +-
.../java/org/apache/beam/sdk/transforms/ParDo.java | 18 +-
.../java/org/apache/beam/sdk/transforms/Reify.java | 4 +-
.../beam/sdk/transforms/ReifyTimestamps.java | 4 +-
.../apache/beam/sdk/transforms/Requirements.java | 2 +-
.../org/apache/beam/sdk/transforms/Reshuffle.java | 18 +-
.../org/apache/beam/sdk/transforms/Sample.java | 14 +-
.../beam/sdk/transforms/SerializableFunctions.java | 7 +-
.../java/org/apache/beam/sdk/transforms/Sum.java | 6 +-
.../java/org/apache/beam/sdk/transforms/Top.java | 2 +-
.../java/org/apache/beam/sdk/transforms/View.java | 24 +-
.../java/org/apache/beam/sdk/transforms/Watch.java | 31 +-
.../org/apache/beam/sdk/transforms/WithKeys.java | 9 +-
.../apache/beam/sdk/transforms/WithTimestamps.java | 2 +-
.../beam/sdk/transforms/display/DisplayData.java | 13 +-
.../beam/sdk/transforms/join/CoGbkResult.java | 11 +-
.../beam/sdk/transforms/join/CoGroupByKey.java | 15 +-
.../sdk/transforms/join/KeyedPCollectionTuple.java | 2 +-
.../reflect/ByteBuddyDoFnInvokerFactory.java | 44 +-
.../reflect/ByteBuddyOnTimerInvokerFactory.java | 66 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 16 +-
.../beam/sdk/transforms/windowing/AfterAll.java | 2 +-
.../beam/sdk/transforms/windowing/AfterEach.java | 2 +-
.../beam/sdk/transforms/windowing/AfterFirst.java | 2 +-
.../transforms/windowing/AfterProcessingTime.java | 2 +-
.../sdk/transforms/windowing/AfterWatermark.java | 7 +-
.../beam/sdk/transforms/windowing/Window.java | 2 +-
.../java/org/apache/beam/sdk/util/ApiSurface.java | 83 +-
.../java/org/apache/beam/sdk/util/ClassPath.java | 8 +-
.../java/org/apache/beam/sdk/util/DoFnInfo.java | 22 +-
.../java/org/apache/beam/sdk/util/Sleeper.java | 7 +-
.../java/org/apache/beam/sdk/util/StreamUtils.java | 2 +-
.../org/apache/beam/sdk/util/WindowedValue.java | 2 +-
.../beam/sdk/util/common/ReflectHelpers.java | 46 +-
.../org/apache/beam/sdk/values/PCollection.java | 4 +-
.../apache/beam/sdk/values/PCollectionList.java | 2 +-
.../apache/beam/sdk/values/PCollectionTuple.java | 4 +-
.../apache/beam/sdk/values/PCollectionViews.java | 9 +-
.../org/apache/beam/sdk/values/TupleTagList.java | 2 +-
.../org/apache/beam/sdk/values/TypeDescriptor.java | 2 +-
.../apache/beam/sdk/values/TypeDescriptors.java | 14 +-
.../apache/beam/sdk/values/ValueWithRecordId.java | 5 +-
.../apache/beam/sdk/values/WindowingStrategy.java | 98 ++-
.../java/org/apache/beam/sdk/PipelineTest.java | 68 +-
.../org/apache/beam/sdk/coders/AvroCoderTest.java | 4 +-
.../apache/beam/sdk/coders/CoderProvidersTest.java | 13 +-
.../apache/beam/sdk/coders/CoderRegistryTest.java | 12 +
.../java/org/apache/beam/sdk/coders/CoderTest.java | 2 +-
.../beam/sdk/coders/CollectionCoderTest.java | 15 +-
.../apache/beam/sdk/coders/CommonCoderTest.java | 2 +-
.../apache/beam/sdk/coders/DefaultCoderTest.java | 9 +-
.../apache/beam/sdk/coders/DelegateCoderTest.java | 36 +-
.../apache/beam/sdk/coders/IterableCoderTest.java | 14 +-
.../org/apache/beam/sdk/coders/KvCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/ListCoderTest.java | 20 +-
.../org/apache/beam/sdk/coders/MapCoderTest.java | 9 +-
.../apache/beam/sdk/coders/NullableCoderTest.java | 16 +-
.../org/apache/beam/sdk/coders/SetCoderTest.java | 9 +-
.../java/org/apache/beam/sdk/io/AvroIOTest.java | 87 +-
.../org/apache/beam/sdk/io/AvroSourceTest.java | 18 +-
.../org/apache/beam/sdk/io/CountingSourceTest.java | 33 +-
.../org/apache/beam/sdk/io/FileBasedSinkTest.java | 27 +-
.../apache/beam/sdk/io/FileBasedSourceTest.java | 30 +-
.../java/org/apache/beam/sdk/io/FileIOTest.java | 51 +-
.../org/apache/beam/sdk/io/FileSystemsTest.java | 14 +-
.../apache/beam/sdk/io/GenerateSequenceTest.java | 19 +-
.../apache/beam/sdk/io/LocalFileSystemTest.java | 37 +-
.../java/org/apache/beam/sdk/io/SimpleSink.java | 7 +-
.../org/apache/beam/sdk/io/TextIOReadTest.java | 20 +-
.../org/apache/beam/sdk/io/TextIOWriteTest.java | 18 +-
.../org/apache/beam/sdk/io/WriteFilesTest.java | 26 +-
.../sdk/options/PipelineOptionsFactoryTest.java | 4 +-
.../sdk/options/PipelineOptionsReflectorTest.java | 5 +-
.../beam/sdk/options/PipelineOptionsTest.java | 2 +-
.../sdk/options/ProxyInvocationHandlerTest.java | 40 +-
.../apache/beam/sdk/options/ValueProviderTest.java | 24 +-
.../beam/sdk/options/ValueProvidersTest.java | 4 +-
.../beam/sdk/runners/PipelineRunnerTest.java | 26 +-
.../beam/sdk/runners/TransformHierarchyTest.java | 42 +-
.../apache/beam/sdk/runners/TransformTreeTest.java | 4 +-
.../sdk/runners/dataflow/TestCountingSource.java | 14 +-
.../apache/beam/sdk/testing/ExpectedLogsTest.java | 22 +-
.../beam/sdk/testing/GatherAllPanesTest.java | 65 +-
.../org/apache/beam/sdk/testing/PAssertTest.java | 78 +-
.../beam/sdk/testing/SerializableMatchersTest.java | 12 +-
.../apache/beam/sdk/testing/StaticWindowsTest.java | 29 +-
.../apache/beam/sdk/testing/TestPipelineTest.java | 10 +-
.../apache/beam/sdk/testing/TestStreamTest.java | 80 +-
.../beam/sdk/testing/WindowSupplierTest.java | 9 +-
.../sdk/transforms/ApproximateQuantilesTest.java | 45 +-
.../beam/sdk/transforms/ApproximateUniqueTest.java | 20 +-
.../apache/beam/sdk/transforms/CombineFnsTest.java | 92 +-
.../apache/beam/sdk/transforms/CombineTest.java | 151 ++--
.../org/apache/beam/sdk/transforms/CountTest.java | 12 +-
.../org/apache/beam/sdk/transforms/CreateTest.java | 32 +-
.../apache/beam/sdk/transforms/DistinctTest.java | 36 +-
.../apache/beam/sdk/transforms/DoFnTesterTest.java | 2 +-
.../beam/sdk/transforms/FlatMapElementsTest.java | 48 +-
.../apache/beam/sdk/transforms/FlattenTest.java | 69 +-
.../apache/beam/sdk/transforms/GroupByKeyTest.java | 205 +++--
.../beam/sdk/transforms/GroupIntoBatchesTest.java | 85 +-
.../org/apache/beam/sdk/transforms/KeysTest.java | 4 +-
.../org/apache/beam/sdk/transforms/KvSwapTest.java | 6 +-
.../org/apache/beam/sdk/transforms/LatestTest.java | 34 +-
.../beam/sdk/transforms/MapElementsTest.java | 68 +-
.../org/apache/beam/sdk/transforms/MeanTest.java | 5 +-
.../beam/sdk/transforms/ParDoLifecycleTest.java | 25 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 388 +++++----
.../apache/beam/sdk/transforms/PartitionTest.java | 2 +-
.../org/apache/beam/sdk/transforms/ReifyTest.java | 21 +-
.../beam/sdk/transforms/ReifyTimestampsTest.java | 15 +-
.../apache/beam/sdk/transforms/ReshuffleTest.java | 130 ++-
.../org/apache/beam/sdk/transforms/SampleTest.java | 56 +-
.../beam/sdk/transforms/SimpleStatsFnsTest.java | 14 +-
.../beam/sdk/transforms/SplittableDoFnTest.java | 61 +-
.../org/apache/beam/sdk/transforms/TopTest.java | 52 +-
.../org/apache/beam/sdk/transforms/ValuesTest.java | 4 +-
.../org/apache/beam/sdk/transforms/ViewTest.java | 959 ++++++++++++---------
.../org/apache/beam/sdk/transforms/WatchTest.java | 162 ++--
.../apache/beam/sdk/transforms/WithKeysTest.java | 6 +-
.../beam/sdk/transforms/WithTimestampsTest.java | 47 +-
.../transforms/display/DisplayDataMatchers.java | 22 +-
.../sdk/transforms/join/CoGbkResultCoderTest.java | 20 +-
.../beam/sdk/transforms/join/CoGbkResultTest.java | 8 +-
.../beam/sdk/transforms/join/CoGroupByKeyTest.java | 112 ++-
.../beam/sdk/transforms/join/UnionCoderTest.java | 13 +-
.../sdk/transforms/reflect/DoFnInvokersTest.java | 13 +-
.../sdk/transforms/reflect/DoFnSignaturesTest.java | 2 +-
.../transforms/windowing/CalendarWindowsTest.java | 2 +-
.../sdk/transforms/windowing/FixedWindowsTest.java | 2 +-
.../sdk/transforms/windowing/SessionsTest.java | 10 +-
.../beam/sdk/transforms/windowing/StubTrigger.java | 2 +-
.../beam/sdk/transforms/windowing/TriggerTest.java | 10 +-
.../beam/sdk/transforms/windowing/WindowTest.java | 102 ++-
.../sdk/transforms/windowing/WindowingTest.java | 12 +-
.../org/apache/beam/sdk/util/ApiSurfaceTest.java | 11 +-
.../BufferedElementCountingOutputStreamTest.java | 2 +-
.../org/apache/beam/sdk/util/NameUtilsTest.java | 7 +-
.../beam/sdk/util/NumberedShardedFileTest.java | 10 +-
.../apache/beam/sdk/util/WindowedValueTest.java | 3 +-
.../beam/sdk/util/common/ReflectHelpersTest.java | 16 +-
.../beam/sdk/values/PCollectionListTest.java | 3 +-
.../beam/sdk/values/PCollectionTupleTest.java | 2 +-
.../org/apache/beam/sdk/values/TupleTagTest.java | 4 +-
.../apache/beam/sdk/values/TypedPValueTest.java | 3 +-
.../sdk/extensions/gcp/options/GcsOptions.java | 8 +-
.../sdk/extensions/gcp/storage/GcsFileSystem.java | 33 +-
.../gcp/storage/GcsFileSystemRegistrar.java | 2 +-
.../java/org/apache/beam/sdk/util/GcsUtil.java | 57 +-
.../beam/sdk/util/RetryHttpRequestInitializer.java | 2 +-
.../sdk/extensions/gcp/options/GcpOptionsTest.java | 2 +-
.../extensions/gcp/storage/GcsFileSystemTest.java | 11 +-
.../java/org/apache/beam/sdk/util/GcsUtilTest.java | 25 +-
.../sdk/util/RetryHttpRequestInitializerTest.java | 30 +-
.../beam/sdk/extensions/joinlibrary/Join.java | 20 +-
.../beam/sdk/extensions/protobuf/ProtoCoder.java | 2 +-
.../extensions/protobuf/ByteStringCoderTest.java | 2 +-
.../sdk/extensions/protobuf/ProtoCoderTest.java | 8 +-
sdks/java/extensions/sketching/pom.xml | 1 +
.../extensions/sketching/ApproximateDistinct.java | 19 +-
.../extensions/sketching/SketchFrequencies.java | 520 +++++++++++
.../sketching/ApproximateDistinctTest.java | 13 +-
.../sketching/SketchFrequenciesTest.java | 204 +++++
.../beam/sdk/extensions/sorter/InMemorySorter.java | 10 +-
.../beam/sdk/extensions/sorter/SortValues.java | 2 +-
.../sorter/BufferedExternalSorterTest.java | 21 +-
.../sdk/extensions/sorter/ExternalSorterTest.java | 11 +-
.../sdk/extensions/sorter/InMemorySorterTest.java | 21 +-
.../beam/sdk/extensions/sorter/SortValuesTest.java | 5 +-
sdks/java/extensions/sql/NOTICE | 2 +-
.../apache/beam/sdk/extensions/sql/BeamSql.java | 3 +-
.../sdk/extensions/sql/example/BeamSqlExample.java | 18 +-
.../date/BeamSqlCurrentDateExpression.java | 2 +-
.../DatetimeReinterpretConversions.java | 31 +-
.../reinterpret/IntegerReinterpretConversions.java | 17 +-
.../operator/reinterpret/Reinterpreter.java | 4 +-
.../sql/impl/planner/BeamQueryPlanner.java | 5 +-
.../sql/impl/rel/BeamAggregationRel.java | 14 +-
.../sdk/extensions/sql/impl/rel/BeamJoinRel.java | 4 +-
.../sql/impl/rel/BeamSetOperatorRelBase.java | 20 +-
.../sdk/extensions/sql/impl/rel/BeamSortRel.java | 19 +-
.../extensions/sql/impl/utils/CalciteUtils.java | 13 +-
.../extensions/sql/BeamSqlDslAggregationTest.java | 16 +-
.../sdk/extensions/sql/BeamSqlDslFilterTest.java | 12 +-
.../sdk/extensions/sql/BeamSqlDslJoinTest.java | 16 +-
.../sdk/extensions/sql/BeamSqlDslProjectTest.java | 20 +-
.../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java | 10 +-
.../sdk/extensions/sql/BeamSqlNonAsciiTest.java | 12 +-
.../impl/interpreter/BeamSqlFnExecutorTest.java | 31 +-
.../sql/impl/schema/BeamSqlRowCoderTest.java | 35 +-
.../transform/BeamAggregationTransformTest.java | 366 ++++----
.../schema/transform/BeamTransformBaseTest.java | 62 +-
.../sql/impl/utils/SqlTypeUtilsTest.java | 13 +-
.../meta/provider/kafka/BeamKafkaCSVTableTest.java | 19 +-
.../meta/provider/text/BeamTextCSVTableTest.java | 17 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 18 +-
.../sdk/fn}/data/BeamFnDataInboundObserver.java | 6 +-
.../fn/data/SettableFutureInboundDataClient.java} | 34 +-
.../sdk/fn/stream/BufferingStreamObserver.java | 9 +-
.../beam/sdk/fn/stream/StreamObserverFactory.java | 7 +-
.../BeamFnDataBufferingOutboundObserverTest.java | 14 +-
.../beam/sdk/fn/stream/AdvancingPhaserTest.java | 10 +-
.../sdk/fn/stream/BufferingStreamObserverTest.java | 62 +-
.../sdk/fn/stream/DirectStreamObserverTest.java | 61 +-
.../ForwardingClientResponseObserverTest.java | 6 +-
.../sdk/fn/stream/StreamObserverFactoryTest.java | 9 +-
.../org/apache/beam/sdk/fn/test/TestExecutors.java | 7 +-
.../apache/beam/sdk/fn/test/TestExecutorsTest.java | 21 +-
.../org/apache/beam/sdk/fn/test/TestStreams.java | 39 +-
.../apache/beam/sdk/fn/test/TestStreamsTest.java | 44 +-
.../org/apache/beam/fn/harness/FlattenRunner.java | 2 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 148 ++--
.../beam/fn/harness/data/BeamFnDataGrpcClient.java | 22 +-
.../beam/fn/harness/BeamFnDataReadRunnerTest.java | 56 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 18 +-
.../org/apache/beam/fn/harness/FnHarnessTest.java | 41 +-
.../harness/control/BeamFnControlClientTest.java | 41 +-
.../harness/control/ProcessBundleHandlerTest.java | 147 ++--
.../fn/harness/control/RegisterHandlerTest.java | 17 +-
.../fn/harness/data/BeamFnDataGrpcClientTest.java | 15 +-
.../data/BeamFnDataInboundObserverTest.java | 1 +
.../harness/logging/BeamFnLoggingClientTest.java | 18 +-
.../harness/state/StateFetchingIteratorsTest.java | 47 +-
.../org/apache/beam/sdk/io/amqp/AmqpIOTest.java | 3 +-
.../apache/beam/sdk/io/cassandra/CassandraIO.java | 8 +-
.../sdk/io/cassandra/CassandraServiceImpl.java | 6 +-
.../beam/sdk/io/cassandra/CassandraIOIT.java | 22 +-
.../beam/sdk/io/cassandra/CassandraIOTest.java | 19 +-
.../org/apache/beam/sdk/io/common/TestRow.java | 2 +-
.../elasticsearch/ElasticsearchIOTestCommon.java | 10 +-
.../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 36 +-
sdks/java/io/file-based-io-tests/pom.xml | 104 +++
.../java/org/apache/beam/sdk/io/avro/AvroIOIT.java | 37 +-
.../java/org/apache/beam/sdk/io/text/TextIOIT.java | 23 +-
.../apache/beam/sdk/io/tfrecord/TFRecordIOIT.java | 20 +-
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 110 ++-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 24 +-
.../sdk/io/gcp/bigquery/BigQueryQuerySource.java | 11 +-
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 63 +-
.../sdk/io/gcp/bigquery/DynamicDestinations.java | 24 +-
.../gcp/bigquery/DynamicDestinationsHelpers.java | 4 +-
.../io/gcp/bigquery/PassThroughThenCleanup.java | 5 +-
.../beam/sdk/io/gcp/bigquery/ReifyAsIterable.java | 2 +-
.../sdk/io/gcp/bigquery/StreamingWriteTables.java | 25 +-
.../beam/sdk/io/gcp/bigquery/WriteResult.java | 2 +-
.../beam/sdk/io/gcp/bigquery/WriteTables.java | 13 +-
.../beam/sdk/io/gcp/bigtable/BigtableConfig.java | 100 ++-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 240 +++---
.../sdk/io/gcp/bigtable/BigtableServiceImpl.java | 43 +-
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 2 +-
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 6 +-
.../gcp/pubsub/PubsubMessagePayloadOnlyCoder.java | 3 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSink.java | 2 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSource.java | 4 +-
.../beam/sdk/io/gcp/spanner/OrderedCode.java | 3 +-
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 41 +-
.../sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java | 15 +-
.../sdk/io/gcp/bigquery/BigQueryIOReadTest.java | 9 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 139 ++-
.../io/gcp/bigquery/BigQueryServicesImplTest.java | 10 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtilTest.java | 25 +-
.../sdk/io/gcp/bigtable/BigtableConfigTest.java | 252 ++++++
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 151 ++--
.../beam/sdk/io/gcp/bigtable/BigtableReadIT.java | 7 +-
.../io/gcp/bigtable/BigtableServiceImplTest.java | 146 ++++
.../beam/sdk/io/gcp/datastore/DatastoreV1Test.java | 29 +-
.../apache/beam/sdk/io/gcp/datastore/V1ReadIT.java | 8 +-
.../sdk/io/gcp/pubsub/PubsubJsonClientTest.java | 4 +-
.../sdk/io/gcp/pubsub/PubsubTestClientTest.java | 13 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSinkTest.java | 26 +-
.../io/gcp/pubsub/PubsubUnboundedSourceTest.java | 9 +-
.../io/gcp/spanner/MutationGroupEncoderTest.java | 9 +-
.../beam/sdk/io/gcp/spanner/SpannerIOReadTest.java | 6 +-
.../beam/sdk/io/gcp/spanner/SpannerReadIT.java | 3 +-
.../apache/beam/sdk/io/hdfs/HadoopFileSystem.java | 3 +-
.../io/hdfs/HadoopFileSystemOptionsRegistrar.java | 2 +-
.../sdk/io/hdfs/HadoopFileSystemOptionsTest.java | 2 +-
.../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 31 +-
.../build.gradle | 27 +-
.../input-format => hadoop-input-format}/pom.xml | 189 +++-
.../io/hadoop/inputformat/HadoopInputFormatIO.java | 62 +-
.../sdk/io/hadoop/inputformat/package-info.java | 0
.../ConfigurableEmployeeInputFormat.java | 2 +-
.../beam/sdk/io/hadoop/inputformat/Employee.java | 0
.../io/hadoop/inputformat/EmployeeInputFormat.java | 2 +-
.../io/hadoop/inputformat}/HIFIOCassandraIT.java | 20 +-
.../sdk/io/hadoop/inputformat}/HIFIOElasticIT.java | 37 +-
.../hadoop/inputformat/HIFIOWithElasticTest.java | 26 +-
.../HIFIOWithEmbeddedCassandraTest.java | 9 +-
.../io/hadoop/inputformat/HIFITestOptions.java} | 4 +-
.../hadoop/inputformat/HadoopInputFormatIOIT.java | 32 +-
.../inputformat/HadoopInputFormatIOTest.java | 42 +-
.../ReuseObjectsEmployeeInputFormat.java | 2 +-
.../io/hadoop/inputformat/TestEmployeeDataSet.java | 16 +-
.../io/hadoop/inputformat/TestRowDBWritable.java | 0
.../src/test/resources/cassandra.yaml | 0
sdks/java/io/hadoop/input-format/build.gradle | 60 --
sdks/java/io/hadoop/jdk1.8-tests/pom.xml | 253 ------
sdks/java/io/hadoop/pom.xml | 82 --
.../org/apache/beam/sdk/io/hbase/HBaseIOTest.java | 8 +-
.../beam/sdk/io/hbase/SerializableScanTest.java | 6 +-
.../beam/sdk/io/hcatalog/HCatalogIOTest.java | 2 +-
.../beam/sdk/io/hcatalog/HCatalogIOTestUtils.java | 2 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 22 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java | 9 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 88 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOTest.java | 4 +-
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 238 +++--
.../apache/beam/sdk/io/kafka/KafkaRecordCoder.java | 6 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 194 ++---
.../sdk/io/kinesis/DynamicCheckpointGenerator.java | 14 +-
.../apache/beam/sdk/io/kinesis/KinesisRecord.java | 4 +-
.../beam/sdk/io/kinesis/ShardReadersPool.java | 26 +-
.../sdk/io/kinesis/SimplifiedKinesisClient.java | 119 ++-
.../beam/sdk/io/kinesis/AmazonKinesisMock.java | 22 +-
.../beam/sdk/io/kinesis/KinesisReaderIT.java | 20 +-
.../beam/sdk/io/kinesis/KinesisReaderTest.java | 27 +-
.../beam/sdk/io/kinesis/KinesisUploader.java | 4 +-
.../beam/sdk/io/kinesis/ShardReadersPoolTest.java | 34 +-
.../sdk/io/kinesis/ShardRecordsIteratorTest.java | 12 +-
.../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 42 +-
.../beam/sdk/io/mongodb/MongoDBGridFSIOTest.java | 131 ++-
.../apache/beam/sdk/io/mongodb/MongoDbIOTest.java | 85 +-
sdks/java/io/pom.xml | 2 +-
.../java/org/apache/beam/sdk/io/redis/RedisIO.java | 13 +-
.../org/apache/beam/sdk/io/redis/RedisIOTest.java | 4 +-
.../java/org/apache/beam/sdk/io/solr/SolrIO.java | 11 +-
.../org/apache/beam/sdk/io/solr/SolrIOTest.java | 7 +-
.../org/apache/beam/sdk/io/tika/TikaIOTest.java | 14 +-
.../java/org/apache/beam/sdk/io/xml/XmlIO.java | 14 +-
.../java/org/apache/beam/sdk/io/xml/XmlSource.java | 7 +-
.../org/apache/beam/sdk/io/xml/JAXBCoderTest.java | 31 +-
.../java/org/apache/beam/sdk/io/xml/XmlIOTest.java | 8 +-
.../org/apache/beam/sdk/io/xml/XmlSourceTest.java | 11 +-
.../maven-archetypes/examples-java8/build.gradle | 33 -
.../examples-java8/generate-sources.sh | 84 --
sdks/java/maven-archetypes/examples-java8/pom.xml | 107 ---
.../META-INF/maven/archetype-metadata.xml | 39 -
.../src/main/resources/archetype-resources/pom.xml | 421 ---------
.../resources/projects/basic/archetype.properties | 21 -
.../src/test/resources/projects/basic/goal.txt | 1 -
.../src/main/resources/archetype-resources/pom.xml | 8 +
sdks/java/maven-archetypes/pom.xml | 15 -
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 71 +-
.../org/apache/beam/sdk/nexmark/model/Bid.java | 43 +-
.../model/sql/adapter/ModelAdaptersMapping.java | 46 +-
.../beam/sdk/nexmark/queries/NexmarkQuery.java | 32 +-
.../apache/beam/sdk/nexmark/queries/Query10.java | 361 ++++----
.../apache/beam/sdk/nexmark/queries/Query11.java | 17 +-
.../apache/beam/sdk/nexmark/queries/Query12.java | 49 +-
.../apache/beam/sdk/nexmark/queries/Query2.java | 14 +-
.../apache/beam/sdk/nexmark/queries/Query3.java | 25 +-
.../apache/beam/sdk/nexmark/queries/Query4.java | 26 +-
.../apache/beam/sdk/nexmark/queries/Query5.java | 5 +-
.../apache/beam/sdk/nexmark/queries/Query6.java | 14 +-
.../apache/beam/sdk/nexmark/queries/Query7.java | 9 +-
.../apache/beam/sdk/nexmark/queries/Query8.java | 25 +-
.../beam/sdk/nexmark/queries/WinningBids.java | 92 +-
.../sdk/nexmark/model/sql/BeamRecordSizeTest.java | 29 +-
.../python/apache_beam/examples/wordcount_fnapi.py | 8 +-
sdks/python/apache_beam/io/filesystem.py | 6 +
sdks/python/apache_beam/io/filesystems.py | 13 +-
.../apache_beam/io/gcp/gcsfilesystem_test.py | 71 +-
sdks/python/apache_beam/io/hadoopfilesystem.py | 4 +-
.../python/apache_beam/io/hadoopfilesystem_test.py | 4 +-
sdks/python/apache_beam/io/localfilesystem_test.py | 4 +-
sdks/python/apache_beam/pipeline.py | 3 +
sdks/python/apache_beam/pipeline_test.py | 15 +-
.../runners/dataflow/dataflow_runner.py | 9 +
.../runners/dataflow/internal/apiclient.py | 2 +-
.../runners/dataflow/native_io/iobase_test.py | 3 +-
.../runners/direct/direct_runner_test.py | 2 +-
.../runners/direct/sdf_direct_runner_test.py | 6 +-
.../apache_beam/runners/worker/bundle_processor.py | 1 -
.../apache_beam/runners/worker/sdk_worker.py | 2 +-
sdks/python/apache_beam/transforms/core.py | 14 +-
.../apache_beam/transforms/ptransform_test.py | 18 +
sdks/python/pom.xml | 6 +-
sdks/python/setup.cfg | 29 +
sdks/python/tox.ini | 21 +
settings.gradle | 4 +-
763 files changed, 11790 insertions(+), 13122 deletions(-)
copy .test-infra/jenkins/{job_beam_Release_NightlySnapshot.groovy => job_beam_PostRelease_NightlySnapshot.groovy} (51%)
copy .test-infra/kubernetes/{postgres/pkb-config-local.yml => hadoop/SmallITCluster/pkb-config.yml} (55%)
copy sdks/java/io/kafka/README.md => CONTRIBUTING.md (60%)
rename examples/{java8 => java}/src/main/java/org/apache/beam/examples/complete/game/GameStats.java (68%)
rename examples/{java8 => java}/src/main/java/org/apache/beam/examples/complete/game/HourlyTeamScore.java (75%)
rename examples/{java8 => java}/src/main/java/org/apache/beam/examples/complete/game/LeaderBoard.java (87%)
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/game/StatefulTeamScore.java
rename examples/{java8 => java}/src/main/java/org/apache/beam/examples/complete/game/UserScore.java (94%)
rename examples/{java8 => java}/src/main/java/org/apache/beam/examples/complete/game/injector/Injector.java (91%)
rename examples/{java8 => java}/src/main/java/org/apache/beam/examples/complete/game/injector/InjectorUtils.java (100%)
rename examples/{java8 => java}/src/main/java/org/apache/beam/examples/complete/game/injector/RetryHttpInitializerWrapper.java (71%)
copy runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/CloudObjectKinds.java => examples/java/src/main/java/org/apache/beam/examples/complete/game/utils/GameConstants.java (60%)
rename examples/{java8 => java}/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToBigQuery.java (100%)
rename examples/{java8 => java}/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java (99%)
rename examples/{java8 => java}/src/main/java/org/apache/beam/examples/complete/game/utils/WriteWindowedToBigQuery.java (100%)
rename examples/{java8 => java}/src/main/java/org/apache/beam/examples/website_snippets/Snippets.java (92%)
rename examples/{java8/src/test/java/org/apache/beam/examples/MinimalWordCountJava8Test.java => java/src/test/java/org/apache/beam/examples/MinimalWordCountTest.java} (61%)
rename examples/{java8 => java}/src/test/java/org/apache/beam/examples/complete/game/GameStatsTest.java (100%)
rename examples/{java8 => java}/src/test/java/org/apache/beam/examples/complete/game/HourlyTeamScoreTest.java (100%)
rename examples/{java8 => java}/src/test/java/org/apache/beam/examples/complete/game/LeaderBoardTest.java (97%)
create mode 100644 examples/java/src/test/java/org/apache/beam/examples/complete/game/StatefulTeamScoreTest.java
rename examples/{java8 => java}/src/test/java/org/apache/beam/examples/complete/game/UserScoreTest.java (100%)
rename examples/{java8 => java}/src/test/java/org/apache/beam/examples/website_snippets/SnippetsTest.java (97%)
delete mode 100644 examples/java8/build.gradle
delete mode 100644 examples/java8/pom.xml
delete mode 100644 examples/java8/src/main/java/org/apache/beam/examples/MinimalWordCountJava8.java
create mode 100644 release/TestScripts.groovy
create mode 100644 release/quickstart-java-direct.groovy
copy runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/{TransformPayloadTranslatorRegistrar.java => Environments.java} (58%)
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/GrpcDataService.java
create mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/data/GrpcDataServiceTest.java
create mode 100644 sdks/java/extensions/sketching/src/main/java/org/apache/beam/sdk/extensions/sketching/SketchFrequencies.java
create mode 100644 sdks/java/extensions/sketching/src/test/java/org/apache/beam/sdk/extensions/sketching/SketchFrequenciesTest.java
rename sdks/java/{harness/src/main/java/org/apache/beam/fn/harness => fn-execution/src/main/java/org/apache/beam/sdk/fn}/data/BeamFnDataInboundObserver.java (93%)
copy sdks/java/{harness/src/main/java/org/apache/beam/fn/harness/data/CompletableFutureInboundDataClient.java => fn-execution/src/main/java/org/apache/beam/sdk/fn/data/SettableFutureInboundDataClient.java} (54%)
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfigTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImplTest.java
rename sdks/java/io/{hadoop/jdk1.8-tests => hadoop-input-format}/build.gradle (86%)
rename sdks/java/io/{hadoop/input-format => hadoop-input-format}/pom.xml (64%)
rename sdks/java/io/{hadoop/input-format => hadoop-input-format}/src/main/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIO.java (96%)
rename sdks/java/io/{hadoop/input-format => hadoop-input-format}/src/main/java/org/apache/beam/sdk/io/hadoop/inputformat/package-info.java (100%)
rename sdks/java/io/{hadoop/input-format => hadoop-input-format}/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/ConfigurableEmployeeInputFormat.java (98%)
rename sdks/java/io/{hadoop/input-format => hadoop-input-format}/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/Employee.java (100%)
rename sdks/java/io/{hadoop/input-format => hadoop-input-format}/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/EmployeeInputFormat.java (98%)
rename sdks/java/io/{hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/integration/tests => hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat}/HIFIOCassandraIT.java (91%)
rename sdks/java/io/{hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/integration/tests => hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat}/HIFIOElasticIT.java (90%)
rename sdks/java/io/{hadoop/jdk1.8-tests => hadoop-input-format}/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithElasticTest.java (95%)
rename sdks/java/io/{hadoop/jdk1.8-tests => hadoop-input-format}/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java (97%)
rename sdks/java/io/{hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/custom/options/HIFTestOptions.java => hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFITestOptions.java} (94%)
rename sdks/java/io/{hadoop/input-format => hadoop-input-format}/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOIT.java (84%)
rename sdks/java/io/{hadoop/input-format => hadoop-input-format}/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java (97%)
rename sdks/java/io/{hadoop/input-format => hadoop-input-format}/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/ReuseObjectsEmployeeInputFormat.java (98%)
rename sdks/java/io/{hadoop/input-format => hadoop-input-format}/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/TestEmployeeDataSet.java (85%)
rename sdks/java/io/{hadoop/input-format => hadoop-input-format}/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/TestRowDBWritable.java (100%)
rename sdks/java/io/{hadoop/jdk1.8-tests => hadoop-input-format}/src/test/resources/cassandra.yaml (100%)
delete mode 100644 sdks/java/io/hadoop/input-format/build.gradle
delete mode 100644 sdks/java/io/hadoop/jdk1.8-tests/pom.xml
delete mode 100644 sdks/java/io/hadoop/pom.xml
delete mode 100644 sdks/java/maven-archetypes/examples-java8/build.gradle
delete mode 100755 sdks/java/maven-archetypes/examples-java8/generate-sources.sh
delete mode 100644 sdks/java/maven-archetypes/examples-java8/pom.xml
delete mode 100644 sdks/java/maven-archetypes/examples-java8/src/main/resources/META-INF/maven/archetype-metadata.xml
delete mode 100644 sdks/java/maven-archetypes/examples-java8/src/main/resources/archetype-resources/pom.xml
delete mode 100644 sdks/java/maven-archetypes/examples-java8/src/test/resources/projects/basic/archetype.properties
delete mode 100644 sdks/java/maven-archetypes/examples-java8/src/test/resources/projects/basic/goal.txt
--
To stop receiving notification emails like this one, please contact
kenn@apache.org.