You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ke...@apache.org on 2017/08/07 23:53:35 UTC
[10/50] [abbrv] beam git commit: This closes #2241: merge master to
gearpump-master and fixup
This closes #2241: merge master to gearpump-master and fixup
[BEAM-79] Fix gearpump-runner merge conflicts and test failure
Revert BigQueryIO bit of 'Make all uses of CountingOutputStream close their resources'
[BEAM-1629] Init metrics/aggregators accumulators before traversing pipeline
Generate zip distribution for pyhthon
Ignore results from the tox clean up phase
Add README to python tarball.
Remove exception suppression from PAssert.SideInputCheckerDoFn
Remove duplicated dependency from Dataflow runner pom.xml
Added assertion failure tests for `PAssert#thatSingleton`
Added a test of default PAssert failure reason
Javadoc changes
[BEAM-1551] Allow `PAssert`s to take a message
add unicode type to the typeDict attribute in Python SDK
Remove Pipeline.getRunner
[BEAM-1686] Use random MQTT clientID when not defined to avoid NPE
Properly deal with late processing-time timers
[BEAM-1661] Shade guava in the JdbcIO
[BEAM-797] A PipelineVisitor that creates a Spark-native pipeline.
Introduce Flink-specific state GC implementations
Move GC timer checking to StatefulDoFnRunner.CleanupTimer
Move pipeline context and add more tests.
Add license to new files.
Runner API translation of triggers and windowing strategies.
Runner API encoding of WindowFns.
Runner API context helper classes.
Auto-generated runner api proto bindings.
HadoopInputFormatIO with junits
Test runner to stop on EOT watermark, or timeout.
[BEAM-1184] Add integration tests to ElasticsearchIO
Jdbc k8 script: postgres data store only accessible inside test project
Jdbc k8 & data loading: add teardown and update names/docs
Bump Dataflow ROS timeout to 120 minutes
Fixup typo in WindowingStrategies
Update archetypes
[BEAM-1649] Fix unresolved references in Python SDK
Upgrade Dataflow container version to beam-master-20170307
Explicitly GBK before stateful ParDo in Dataflow batch
Add ServicesResourceTransformer to all shading configuration
Fix typo in proto: widow -> window.
[BEAM-1646] Remove duplicated bigquery dependency
Update python SDK version to the next version.
Change Json parsing from gson to jackson for ElasticsearchIO
[maven-release-plugin] prepare for next development iteration
[maven-release-plugin] prepare branch release-0.6.0
[BEAM-1633] Move .tox/ directory under target/ in Python SDK
Make all uses of CountingOutputStream close their resources
Flink: register known IOChannelFactories
[BEAM-1546] Specify exact version for Python in the SDK
[BEAM-1635] TypeError in AfterWatermark class's __repr__ method
[BEAM-1636] UnboundedDataset action() does not materialize RDD
[BEAM-1556] Make PipelineOptions a lazy-singleton and init IOs as part of it.
Add tests for serialization of BigQueryIO.TableRowInfoCoder
Fix tox warning for non-whitelisted find command
Revert "Implement Single-Output ParDo as a composite"
[BEAM-1623] Transform Reshuffle directly in Spark runner
[BEAM-1626] Remove cache of MapWithStateDStream on read.
Revert "DataflowRunner: experimental support for issuing FnAPI based jobs"
Java examples: move shade plugin into default lifecycle
Java 8 examples: add shade configuration for producing bundled/fat jar
Java 8 examples: add SNAPSHOT repository to the pom.xml file
Update Guava version from 19.0 to 20.0 in example projects
[BEAM-1625] BoundedDataset action() does not materialize RDD
Revert "[maven-release-plugin] prepare branch release-0.6.0"
Updating Dataflow API client protobufs
Implement Single-Output ParDo as a composite
Add a Test for windowed CombineGloballyAsSingletonView
Remove SingletonCombine
Only Override CreatePCollectionView in Streaming
[BEAM-1310] Add running integration tests in JdbcIO on Spark and Dataflow runners
DataflowRunner: experimental support for issuing FnAPI based jobs
Updates Python SDK source API so that sources can report limited parallelism signals.
[BEAM-1188] Python Bigquery Verifier For E2E Test
Deprecate Pipeline.getOptions
BEAM-1567 hashStream should be closed in PackageUtil#createPackageAttributes()
[BEAM-1565] Update Spark runner PostCommit Jenkins job.
[maven-release-plugin] prepare branch release-0.6.0
Do not Reassign Windows when WindowFn is null
Fix DataflowRunner message about uploaded vs cached files
Update javadoc ant to include runners/ and exclude modules with a wildcard
Adding per-stage matching to metrics filters
Upgrade dill to 0.2.6 and pin it
Remove PipelineRunner#apply
[BEAM-111] Move WritableCoder to hadoop-common
[BEAM-1297] Update maven shade plugin, fix typo and remove unneeded version
[BEAM-351] Add DisplayData to KafkaIO
Inline rather than reference FunctionSpecs.
[BEAM-1517] Garbage collect user state in Flink Runner
BEAM-1417 Count should comply with PTransform style guide
BEAM-1419 Flatten should comply with PTransform style guide
BEAM-1416 Write transform should comply with PTransform style guide
BEAM-1426 SortValues should comply with PTransform style guide
BEAM-1424 ToString should comply with PTransform style guide
BEAM-1423 Sample should comply with PTransform style guide
BEAM-1421 Latest should comply with PTransform style guide
BEAM-1420 GroupByKey should comply with PTransform style guide
Use UnsupportedSideInputReader in GroupAlsoByWindowEvaluatorFactory
Include cython tests in presubmits for linux platform
Update output stream cython declaration
Make side inputs a map, rather than embedding the name in the message.
Streaming tests, especially the ones using checkpoints, need a time buffer to finish.
...
Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/555842a1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/555842a1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/555842a1
Branch: refs/heads/master
Commit: 555842a1a0c478b3935b3987683d2645eba770c3
Parents: 15a8ad6 3eab6a6
Author: Kenneth Knowles <kl...@google.com>
Authored: Fri Mar 17 11:15:52 2017 -0700
Committer: Kenneth Knowles <kl...@google.com>
Committed: Fri Mar 17 11:15:52 2017 -0700
----------------------------------------------------------------------
.gitignore | 10 +
.jenkins/common_job_properties.groovy | 98 +-
...job_beam_PostCommit_Java_MavenInstall.groovy | 4 +-
...ostCommit_Java_RunnableOnService_Apex.groovy | 8 +-
...ommit_Java_RunnableOnService_Dataflow.groovy | 8 +-
...stCommit_Java_RunnableOnService_Flink.groovy | 8 +-
...ommit_Java_RunnableOnService_Gearpump.groovy | 12 +-
...stCommit_Java_RunnableOnService_Spark.groovy | 10 +-
.../job_beam_PostCommit_Python_Verify.groovy | 8 +-
.../job_beam_PreCommit_Java_MavenInstall.groovy | 6 +-
.../job_beam_PreCommit_Website_Stage.groovy | 80 +
.jenkins/job_beam_PreCommit_Website_Test.groovy | 65 +
.../job_beam_Release_NightlySnapshot.groovy | 9 +-
.jenkins/job_seed.groovy | 24 +-
.travis.yml | 24 +-
.travis/README.md | 2 +-
DISCLAIMER | 10 -
NOTICE | 4 +-
README.md | 57 +-
examples/java/README.md | 16 +-
examples/java/pom.xml | 24 +-
.../beam/examples/DebuggingWordCount.java | 4 +-
.../org/apache/beam/examples/WordCount.java | 6 +-
.../beam/examples/complete/AutoComplete.java | 4 +-
.../org/apache/beam/examples/complete/README.md | 14 +-
.../apache/beam/examples/complete/TfIdf.java | 2 +-
.../examples/complete/TopWikipediaSessions.java | 27 +-
.../examples/complete/TrafficMaxLaneFlow.java | 2 +-
.../beam/examples/complete/TrafficRoutes.java | 2 +-
.../examples/cookbook/BigQueryTornadoes.java | 2 +-
.../cookbook/CombinePerKeyExamples.java | 2 +-
.../org/apache/beam/examples/cookbook/README.md | 14 +-
.../beam/examples/cookbook/TriggerExample.java | 4 +-
.../beam/examples/WindowedWordCountIT.java | 16 +-
.../org/apache/beam/examples/WordCountTest.java | 7 +-
.../examples/complete/AutoCompleteTest.java | 11 +-
.../beam/examples/complete/TfIdfTest.java | 6 +-
.../complete/TopWikipediaSessionsTest.java | 7 +-
.../examples/cookbook/DistinctExampleTest.java | 9 +-
.../examples/cookbook/JoinExamplesTest.java | 6 +-
.../examples/cookbook/TriggerExampleTest.java | 6 +-
examples/java8/pom.xml | 3 +-
.../beam/examples/complete/game/GameStats.java | 7 +-
.../examples/complete/game/LeaderBoard.java | 5 +-
.../beam/examples/complete/game/UserScore.java | 2 +-
.../examples/MinimalWordCountJava8Test.java | 6 +-
.../examples/complete/game/GameStatsTest.java | 7 +-
.../complete/game/HourlyTeamScoreTest.java | 5 +-
.../examples/complete/game/LeaderBoardTest.java | 11 +-
.../examples/complete/game/UserScoreTest.java | 10 +-
examples/pom.xml | 16 +-
pom.xml | 319 +-
runners/apex/README.md | 4 +-
runners/apex/pom.xml | 32 +-
.../beam/runners/apex/ApexPipelineOptions.java | 7 +-
.../apache/beam/runners/apex/ApexRunner.java | 144 +-
.../beam/runners/apex/ApexYarnLauncher.java | 29 +-
.../beam/runners/apex/TestApexRunner.java | 10 -
.../translation/ApexPipelineTranslator.java | 17 +-
.../translation/CreateValuesTranslator.java | 25 +-
.../FlattenPCollectionTranslator.java | 37 +-
.../apex/translation/GroupByKeyTranslator.java | 2 +-
.../translation/ParDoBoundMultiTranslator.java | 32 +-
.../apex/translation/ParDoBoundTranslator.java | 9 +-
.../apex/translation/TranslationContext.java | 43 +-
.../translation/WindowAssignTranslator.java | 78 +
.../apex/translation/WindowBoundTranslator.java | 78 -
.../operators/ApexFlattenOperator.java | 4 +-
.../operators/ApexGroupByKeyOperator.java | 22 +-
.../operators/ApexParDoOperator.java | 12 +-
.../ApexReadUnboundedInputOperator.java | 17 +-
.../translation/utils/ApexStateInternals.java | 30 +-
.../apex/translation/utils/NoOpStepContext.java | 7 +-
.../beam/runners/apex/ApexRunnerTest.java | 76 +
.../beam/runners/apex/ApexYarnLauncherTest.java | 9 +-
.../runners/apex/examples/WordCountTest.java | 2 +-
.../FlattenPCollectionTranslatorTest.java | 24 +-
.../translation/ParDoBoundTranslatorTest.java | 38 +-
.../translation/ReadUnboundTranslatorTest.java | 8 +-
.../utils/ApexStateInternalsTest.java | 14 +-
.../test/resources/beam-runners-apex.properties | 20 +
runners/core-construction-java/pom.xml | 138 +
.../EmptyFlattenAsCreateFactory.java | 71 +
.../core/construction/PTransformMatchers.java | 192 +
.../core/construction/PrimitiveCreate.java | 77 +
.../core/construction/ReplacementOutputs.java | 105 +
.../SingleInputOutputOverrideFactory.java | 50 +
.../UnsupportedOverrideFactory.java | 71 +
.../runners/core/construction/package-info.java | 22 +
.../construction/PTransformMatchersTest.java | 425 ++
.../construction/ReplacementOutputsTest.java | 254 +
.../SingleInputOutputOverrideFactoryTest.java | 114 +
.../UnsupportedOverrideFactoryTest.java | 65 +
runners/core-java/pom.xml | 24 +-
.../beam/runners/core/AggregatorFactory.java | 1 -
.../beam/runners/core/AssignWindowsDoFn.java | 3 +-
.../beam/runners/core/BaseExecutionContext.java | 174 +
.../apache/beam/runners/core/DoFnAdapters.java | 323 ++
.../apache/beam/runners/core/DoFnRunner.java | 21 -
.../apache/beam/runners/core/DoFnRunners.java | 158 +-
.../beam/runners/core/ExecutionContext.java | 100 +
.../GroupAlsoByWindowViaOutputBufferDoFn.java | 114 +
.../core/GroupAlsoByWindowViaWindowSetDoFn.java | 17 +-
.../GroupAlsoByWindowViaWindowSetNewDoFn.java | 154 +
.../runners/core/GroupAlsoByWindowsDoFn.java | 5 +-
.../GroupAlsoByWindowsViaOutputBufferDoFn.java | 134 -
.../runners/core/InMemoryStateInternals.java | 643 +++
.../runners/core/InMemoryTimerInternals.java | 300 ++
.../apache/beam/runners/core/KeyedWorkItem.java | 2 +-
.../beam/runners/core/KeyedWorkItemCoder.java | 8 +-
.../beam/runners/core/KeyedWorkItems.java | 2 +-
.../core/LateDataDroppingDoFnRunner.java | 2 -
.../apache/beam/runners/core/LateDataUtils.java | 88 +
.../runners/core/MergingActiveWindowSet.java | 4 -
.../beam/runners/core/MergingStateAccessor.java | 41 +
.../apache/beam/runners/core/NonEmptyPanes.java | 7 +-
.../org/apache/beam/runners/core/OldDoFn.java | 471 ++
...eBoundedSplittableProcessElementInvoker.java | 285 ++
.../beam/runners/core/PaneInfoTracker.java | 4 -
.../runners/core/PerKeyCombineFnRunner.java | 70 -
.../runners/core/PerKeyCombineFnRunners.java | 101 -
.../org/apache/beam/runners/core/ReduceFn.java | 2 -
.../runners/core/ReduceFnContextFactory.java | 11 +-
.../beam/runners/core/ReduceFnRunner.java | 6 +-
.../beam/runners/core/SideInputHandler.java | 4 -
.../beam/runners/core/SimpleDoFnRunner.java | 190 +-
.../beam/runners/core/SimpleOldDoFnRunner.java | 12 +-
.../beam/runners/core/SplittableParDo.java | 233 +-
.../core/SplittableProcessElementInvoker.java | 65 +
.../apache/beam/runners/core/StateAccessor.java | 38 +
.../beam/runners/core/StateInternals.java | 59 +
.../runners/core/StateInternalsFactory.java | 35 +
.../apache/beam/runners/core/StateMerging.java | 309 ++
.../beam/runners/core/StateNamespace.java | 56 +
.../runners/core/StateNamespaceForTest.java | 65 +
.../beam/runners/core/StateNamespaces.java | 278 ++
.../apache/beam/runners/core/StateTable.java | 84 +
.../org/apache/beam/runners/core/StateTag.java | 125 +
.../org/apache/beam/runners/core/StateTags.java | 382 ++
.../beam/runners/core/StatefulDoFnRunner.java | 171 +
.../beam/runners/core/SystemReduceFn.java | 5 -
.../core/TestInMemoryStateInternals.java | 63 +
.../beam/runners/core/TimerInternals.java | 286 ++
.../runners/core/TimerInternalsFactory.java | 35 +
.../core/UnboundedReadFromBoundedSource.java | 14 +-
.../core/UnsupportedSideInputReader.java | 52 +
.../apache/beam/runners/core/WatermarkHold.java | 9 +-
.../beam/runners/core/WindowingInternals.java | 81 +
.../core/WindowingInternalsAdapters.java | 1 -
.../core/triggers/AfterAllStateMachine.java | 2 +-
.../AfterDelayFromFirstElementStateMachine.java | 16 +-
.../core/triggers/AfterFirstStateMachine.java | 2 +-
.../core/triggers/AfterPaneStateMachine.java | 12 +-
.../AfterProcessingTimeStateMachine.java | 2 +
...rSynchronizedProcessingTimeStateMachine.java | 7 +-
.../triggers/AfterWatermarkStateMachine.java | 15 +-
.../core/triggers/OrFinallyStateMachine.java | 2 +-
.../triggers/ReshuffleTriggerStateMachine.java | 6 +-
.../core/triggers/TriggerStateMachine.java | 6 +-
.../TriggerStateMachineContextFactory.java | 12 +-
.../triggers/TriggerStateMachineRunner.java | 8 +-
.../core/triggers/TriggerStateMachines.java | 250 +-
.../core/DoFnDelegatingAggregatorTest.java | 144 +
...roupAlsoByWindowViaOutputBufferDoFnTest.java | 109 +
.../core/GroupAlsoByWindowsProperties.java | 7 +-
...oupAlsoByWindowsViaOutputBufferDoFnTest.java | 110 -
.../core/InMemoryStateInternalsTest.java | 574 +++
.../core/InMemoryTimerInternalsTest.java | 195 +
.../runners/core/KeyedWorkItemCoderTest.java | 9 +-
.../core/LateDataDroppingDoFnRunnerTest.java | 3 +-
.../core/MergingActiveWindowSetTest.java | 2 -
.../apache/beam/runners/core/NoOpOldDoFn.java | 72 +
.../beam/runners/core/OldDoFnContextTest.java | 72 +
.../apache/beam/runners/core/OldDoFnTest.java | 192 +
...ndedSplittableProcessElementInvokerTest.java | 146 +
.../core/PushbackSideInputDoFnRunnerTest.java | 8 +-
.../beam/runners/core/ReduceFnRunnerTest.java | 12 +-
.../beam/runners/core/ReduceFnTester.java | 19 +-
.../beam/runners/core/SideInputHandlerTest.java | 1 -
.../beam/runners/core/SimpleDoFnRunnerTest.java | 6 +-
.../runners/core/SimpleOldDoFnRunnerTest.java | 4 +-
.../beam/runners/core/SplittableParDoTest.java | 254 +-
.../beam/runners/core/StateNamespacesTest.java | 130 +
.../apache/beam/runners/core/StateTagTest.java | 205 +
.../runners/core/StatefulDoFnRunnerTest.java | 347 ++
.../beam/runners/core/TimerInternalsTest.java | 105 +
.../UnboundedReadFromBoundedSourceTest.java | 18 +-
.../beam/runners/core/WindowMatchers.java | 3 +-
...chronizedProcessingTimeStateMachineTest.java | 3 +-
.../AfterWatermarkStateMachineTest.java | 19 +
.../core/triggers/NeverStateMachineTest.java | 1 -
.../ReshuffleTriggerStateMachineTest.java | 8 +-
.../triggers/TriggerStateMachineTester.java | 16 +-
.../core/triggers/TriggerStateMachinesTest.java | 167 +-
runners/direct-java/pom.xml | 27 +-
.../runners/direct/AggregatorContainer.java | 2 +-
.../direct/BoundedReadEvaluatorFactory.java | 10 +-
.../CopyOnAccessInMemoryStateInternals.java | 68 +-
.../runners/direct/DirectExecutionContext.java | 8 +-
...ectGBKIntoKeyedWorkItemsOverrideFactory.java | 4 +-
.../beam/runners/direct/DirectGraphVisitor.java | 33 +-
.../direct/DirectGroupByKeyOverrideFactory.java | 3 +-
.../beam/runners/direct/DirectMetrics.java | 31 +-
.../beam/runners/direct/DirectRunner.java | 114 +-
.../runners/direct/DirectTimerInternals.java | 14 +-
...ecycleManagerRemovingTransformEvaluator.java | 19 +-
.../beam/runners/direct/EmptyInputProvider.java | 4 +-
.../beam/runners/direct/EvaluationContext.java | 40 +-
.../direct/ExecutorServiceParallelExecutor.java | 14 +-
.../runners/direct/FlattenEvaluatorFactory.java | 8 +-
.../GroupAlsoByWindowEvaluatorFactory.java | 38 +-
.../direct/GroupByKeyOnlyEvaluatorFactory.java | 10 +-
.../direct/ImmutableListBundleFactory.java | 6 +
.../direct/KeyedPValueTrackingVisitor.java | 52 +-
.../beam/runners/direct/ParDoEvaluator.java | 29 +-
.../runners/direct/ParDoEvaluatorFactory.java | 25 +-
.../direct/ParDoMultiOverrideFactory.java | 154 +-
.../ParDoSingleViaMultiOverrideFactory.java | 5 +-
.../runners/direct/RootProviderRegistry.java | 4 +-
...littableProcessElementsEvaluatorFactory.java | 84 +-
.../direct/StatefulParDoEvaluatorFactory.java | 72 +-
.../direct/TestStreamEvaluatorFactory.java | 44 +-
.../direct/TransformEvaluatorRegistry.java | 6 +-
.../direct/UnboundedReadEvaluatorFactory.java | 22 +-
.../runners/direct/ViewEvaluatorFactory.java | 16 +-
.../beam/runners/direct/WatermarkManager.java | 107 +-
.../runners/direct/WindowEvaluatorFactory.java | 17 +-
.../direct/WriteWithShardingFactory.java | 148 +-
.../runners/direct/AggregatorContainerTest.java | 18 +-
.../direct/BoundedReadEvaluatorFactoryTest.java | 18 +-
.../direct/CloningBundleFactoryTest.java | 16 +-
.../runners/direct/CommittedResultTest.java | 23 +-
.../CopyOnAccessInMemoryStateInternalsTest.java | 79 +-
.../runners/direct/DirectGraphVisitorTest.java | 55 +-
.../DirectGroupByKeyOverrideFactoryTest.java | 51 +
.../beam/runners/direct/DirectMetricsTest.java | 128 +-
.../beam/runners/direct/DirectRunnerTest.java | 8 +-
.../direct/DirectTimerInternalsTest.java | 4 +-
...leManagerRemovingTransformEvaluatorTest.java | 103 +-
.../runners/direct/EvaluationContextTest.java | 30 +-
.../direct/FlattenEvaluatorFactoryTest.java | 8 +-
.../direct/GroupByKeyEvaluatorFactoryTest.java | 5 +-
.../GroupByKeyOnlyEvaluatorFactoryTest.java | 5 +-
.../ImmutabilityCheckingBundleFactoryTest.java | 6 +-
.../ImmutabilityEnforcementFactoryTest.java | 3 +-
.../direct/ImmutableListBundleFactoryTest.java | 37 +-
.../direct/KeyedPValueTrackingVisitorTest.java | 167 +-
.../beam/runners/direct/ParDoEvaluatorTest.java | 9 +-
.../direct/ParDoMultiOverrideFactoryTest.java | 45 +
.../ParDoSingleViaMultiOverrideFactoryTest.java | 46 +
.../runners/direct/SideInputContainerTest.java | 5 +-
.../StatefulParDoEvaluatorFactoryTest.java | 129 +-
.../runners/direct/StepTransformResultTest.java | 5 +-
.../direct/TestStreamEvaluatorFactoryTest.java | 38 +-
.../runners/direct/TransformExecutorTest.java | 4 +-
.../UnboundedReadEvaluatorFactoryTest.java | 22 +-
.../direct/ViewEvaluatorFactoryTest.java | 18 +-
.../direct/WatermarkCallbackExecutorTest.java | 5 +-
.../runners/direct/WatermarkManagerTest.java | 37 +-
.../direct/WindowEvaluatorFactoryTest.java | 87 +-
.../direct/WriteWithShardingFactoryTest.java | 154 +-
runners/flink/README.md | 250 -
runners/flink/examples/pom.xml | 2 +-
.../beam/runners/flink/examples/WordCount.java | 2 +-
.../flink/examples/streaming/AutoComplete.java | 4 +-
.../examples/streaming/KafkaIOExamples.java | 4 +-
.../KafkaWindowedWordCountExample.java | 2 +-
.../examples/streaming/WindowedWordCount.java | 2 +-
runners/flink/pom.xml | 4 +-
runners/flink/runner/pom.xml | 56 +-
.../flink/DefaultParallelismFactory.java | 3 +-
.../flink/FlinkBatchPipelineTranslator.java | 139 +
.../flink/FlinkBatchTransformTranslators.java | 797 ++++
.../flink/FlinkBatchTranslationContext.java | 154 +
.../FlinkPipelineExecutionEnvironment.java | 11 +-
.../runners/flink/FlinkPipelineOptions.java | 6 +-
.../runners/flink/FlinkPipelineTranslator.java | 53 +
.../apache/beam/runners/flink/FlinkRunner.java | 395 +-
.../flink/FlinkStreamingPipelineTranslator.java | 230 +
.../FlinkStreamingTransformTranslators.java | 1043 +++++
.../flink/FlinkStreamingTranslationContext.java | 130 +
.../flink/FlinkStreamingViewOverrides.java | 372 ++
.../flink/PipelineTranslationOptimizer.java | 72 +
.../beam/runners/flink/TestFlinkRunner.java | 24 +-
.../beam/runners/flink/TranslationMode.java | 31 +
.../FlinkBatchPipelineTranslator.java | 143 -
.../FlinkBatchTransformTranslators.java | 745 ---
.../FlinkBatchTranslationContext.java | 145 -
.../translation/FlinkPipelineTranslator.java | 53 -
.../FlinkStreamingPipelineTranslator.java | 154 -
.../FlinkStreamingTransformTranslators.java | 978 ----
.../FlinkStreamingTranslationContext.java | 110 -
.../PipelineTranslationOptimizer.java | 73 -
.../flink/translation/TranslationMode.java | 31 -
.../functions/FlinkAggregatorFactory.java | 53 +
.../functions/FlinkAssignWindows.java | 2 +-
.../functions/FlinkDoFnFunction.java | 136 +-
.../FlinkMergingNonShuffleReduceFunction.java | 57 +-
.../FlinkMergingPartialReduceFunction.java | 37 +-
.../functions/FlinkMergingReduceFunction.java | 31 +-
.../functions/FlinkMultiOutputDoFnFunction.java | 126 -
.../FlinkMultiOutputProcessContext.java | 118 -
.../FlinkMultiOutputPruningFunction.java | 2 +-
.../functions/FlinkNoElementAssignContext.java | 68 -
.../functions/FlinkNoOpStepContext.java | 73 +
.../functions/FlinkPartialReduceFunction.java | 45 +-
.../functions/FlinkProcessContextBase.java | 267 --
.../functions/FlinkReduceFunction.java | 41 +-
.../functions/FlinkSideInputReader.java | 80 +
.../FlinkSingleOutputProcessContext.java | 69 -
.../functions/FlinkStatefulDoFnFunction.java | 198 +
.../utils/SerializedPipelineOptions.java | 4 +-
.../wrappers/streaming/DoFnOperator.java | 671 ++-
.../wrappers/streaming/FlinkStateInternals.java | 1037 -----
.../streaming/KvToByteBufferKeySelector.java | 56 +
.../streaming/SingletonKeyedWorkItem.java | 2 +-
.../streaming/SingletonKeyedWorkItemCoder.java | 10 +-
.../wrappers/streaming/WindowDoFnOperator.java | 457 +-
.../streaming/io/BoundedSourceWrapper.java | 7 +-
.../streaming/io/UnboundedFlinkSink.java | 6 +
.../streaming/io/UnboundedSourceWrapper.java | 159 +-
.../state/FlinkBroadcastStateInternals.java | 865 ++++
.../state/FlinkKeyGroupStateInternals.java | 487 ++
.../state/FlinkSplitStateInternals.java | 260 ++
.../streaming/state/FlinkStateInternals.java | 1053 +++++
.../state/KeyGroupCheckpointedOperator.java | 35 +
.../state/KeyGroupRestoringOperator.java | 32 +
.../wrappers/streaming/state/package-info.java | 22 +
.../beam/runners/flink/PipelineOptionsTest.java | 42 +-
.../beam/runners/flink/WriteSinkITCase.java | 5 +-
.../flink/streaming/DoFnOperatorTest.java | 337 +-
.../FlinkBroadcastStateInternalsTest.java | 245 +
.../FlinkKeyGroupStateInternalsTest.java | 262 ++
.../streaming/FlinkSplitStateInternalsTest.java | 101 +
.../streaming/FlinkStateInternalsTest.java | 39 +-
.../streaming/UnboundedSourceWrapperTest.java | 570 ++-
runners/gearpump/pom.xml | 51 +-
.../gearpump/GearpumpPipelineResult.java | 21 +-
.../gearpump/GearpumpPipelineTranslator.java | 388 +-
.../beam/runners/gearpump/GearpumpRunner.java | 376 +-
.../runners/gearpump/TestGearpumpRunner.java | 38 +-
.../gearpump/examples/StreamingWordCount.java | 98 -
.../gearpump/examples/UnboundedTextSource.java | 139 -
.../runners/gearpump/examples/package-info.java | 22 -
...CreateGearpumpPCollectionViewTranslator.java | 14 +-
.../CreatePCollectionViewTranslator.java | 6 +-
.../translators/CreateValuesTranslator.java | 51 -
.../FlattenPCollectionTranslator.java | 84 -
.../FlattenPCollectionsTranslator.java | 83 +
.../translators/GroupByKeyTranslator.java | 4 +-
.../translators/ParDoBoundMultiTranslator.java | 32 +-
.../translators/ParDoBoundTranslator.java | 7 +-
.../translators/ReadBoundedTranslator.java | 4 +-
.../translators/ReadUnboundedTranslator.java | 4 +-
.../translators/TransformTranslator.java | 2 +-
.../translators/TranslationContext.java | 29 +-
.../translators/WindowAssignTranslator.java | 100 +
.../translators/WindowBoundTranslator.java | 100 -
.../translators/functions/DoFnFunction.java | 12 +-
.../translators/io/UnboundedSourceWrapper.java | 1 +
.../translators/utils/DoFnRunnerFactory.java | 4 +-
.../utils/NoOpAggregatorFactory.java | 2 +-
.../translators/utils/NoOpStepContext.java | 6 +-
.../translators/utils/TranslatorUtils.java | 2 -
.../translators/utils/TranslatorUtilsTest.java | 1 -
runners/google-cloud-dataflow-java/pom.xml | 40 +-
.../beam/runners/dataflow/AssignWindows.java | 89 +
.../dataflow/BatchStatefulParDoOverrides.java | 283 ++
.../runners/dataflow/BatchViewOverrides.java | 1391 ++++++
.../dataflow/DataflowAggregatorTransforms.java | 79 +
.../beam/runners/dataflow/DataflowClient.java | 44 +-
.../dataflow/DataflowMetricUpdateExtractor.java | 109 +
.../runners/dataflow/DataflowPipelineJob.java | 4 +-
.../dataflow/DataflowPipelineTranslator.java | 606 +--
.../beam/runners/dataflow/DataflowRunner.java | 2466 ++--------
.../DataflowUnboundedReadFromBoundedSource.java | 547 +++
.../beam/runners/dataflow/ReadTranslator.java | 102 +
.../dataflow/StreamingViewOverrides.java | 110 +
.../runners/dataflow/TransformTranslator.java | 131 +
.../dataflow/internal/AssignWindows.java | 89 -
.../dataflow/internal/CustomSources.java | 12 +-
.../internal/DataflowAggregatorTransforms.java | 79 -
.../internal/DataflowMetricUpdateExtractor.java | 109 -
.../DataflowUnboundedReadFromBoundedSource.java | 556 ---
.../runners/dataflow/internal/IsmFormat.java | 20 +-
.../dataflow/internal/ReadTranslator.java | 107 -
.../options/DataflowPipelineOptions.java | 36 +-
.../DataflowPipelineWorkerPoolOptions.java | 16 +-
.../dataflow/testing/TestDataflowRunner.java | 17 +-
.../beam/runners/dataflow/util/DoFnInfo.java | 75 +-
.../beam/runners/dataflow/util/GcsStager.java | 18 +-
.../beam/runners/dataflow/util/PackageUtil.java | 363 +-
.../beam/runners/dataflow/dataflow.properties | 6 +-
.../BatchStatefulParDoOverridesTest.java | 169 +
.../dataflow/BatchViewOverridesTest.java | 633 +++
.../dataflow/DataflowPipelineJobTest.java | 184 +-
.../DataflowPipelineTranslatorTest.java | 145 +-
.../runners/dataflow/DataflowRunnerTest.java | 614 +--
...aflowUnboundedReadFromBoundedSourceTest.java | 79 +
...aflowUnboundedReadFromBoundedSourceTest.java | 83 -
.../DataflowPipelineDebugOptionsTest.java | 2 +-
.../options/DataflowPipelineOptionsTest.java | 20 +-
.../options/DataflowProfilingOptionsTest.java | 4 +-
.../testing/TestDataflowRunnerTest.java | 53 +-
.../runners/dataflow/util/PackageUtilTest.java | 99 +-
runners/pom.xml | 17 +-
runners/spark/README.md | 8 +-
runners/spark/pom.xml | 70 +-
.../spark/SparkNativePipelineVisitor.java | 198 +
.../runners/spark/SparkPipelineOptions.java | 6 +-
.../beam/runners/spark/SparkPipelineResult.java | 67 +-
.../apache/beam/runners/spark/SparkRunner.java | 124 +-
.../beam/runners/spark/SparkRunnerDebugger.java | 137 +
.../runners/spark/TestSparkPipelineOptions.java | 61 +
.../beam/runners/spark/TestSparkRunner.java | 204 +-
.../spark/aggregators/AccumulatorSingleton.java | 53 -
.../aggregators/AggregatorsAccumulator.java | 131 +
.../spark/aggregators/NamedAggregators.java | 12 +-
.../spark/aggregators/SparkAggregators.java | 26 +-
.../aggregators/metrics/AggregatorMetric.java | 44 -
.../metrics/AggregatorMetricSource.java | 50 -
.../metrics/WithNamedAggregatorsSupport.java | 174 -
.../spark/aggregators/metrics/sink/CsvSink.java | 39 -
.../aggregators/metrics/sink/GraphiteSink.java | 39 -
.../aggregators/metrics/sink/package-info.java | 23 -
.../coders/BeamSparkRunnerRegistrator.java | 48 +-
.../beam/runners/spark/coders/CoderHelpers.java | 23 +
.../runners/spark/coders/NullWritableCoder.java | 76 -
.../spark/coders/StatelessJavaSerializer.java | 97 +
.../runners/spark/coders/WritableCoder.java | 122 -
.../beam/runners/spark/examples/WordCount.java | 2 +-
.../apache/beam/runners/spark/io/ConsoleIO.java | 4 +-
.../beam/runners/spark/io/CreateStream.java | 198 +-
.../beam/runners/spark/io/MicrobatchSource.java | 9 +-
.../runners/spark/io/SparkUnboundedSource.java | 166 +-
.../runners/spark/metrics/AggregatorMetric.java | 43 +
.../spark/metrics/AggregatorMetricSource.java | 51 +
.../runners/spark/metrics/CompositeSource.java | 49 +
.../spark/metrics/MetricsAccumulator.java | 132 +
.../spark/metrics/MetricsAccumulatorParam.java | 42 +
.../runners/spark/metrics/SparkBeamMetric.java | 69 +
.../spark/metrics/SparkBeamMetricSource.java | 51 +
.../spark/metrics/SparkMetricResults.java | 181 +
.../spark/metrics/SparkMetricsContainer.java | 147 +
.../spark/metrics/WithMetricsSupport.java | 209 +
.../runners/spark/metrics/package-info.java | 20 +
.../runners/spark/metrics/sink/CsvSink.java | 38 +
.../spark/metrics/sink/GraphiteSink.java | 38 +
.../spark/metrics/sink/package-info.java | 22 +
.../SparkGroupAlsoByWindowViaWindowSet.java | 431 ++
.../spark/stateful/SparkStateInternals.java | 418 ++
.../spark/stateful/SparkTimerInternals.java | 193 +
.../spark/stateful/StateSpecFunctions.java | 60 +-
.../spark/translation/BoundedDataset.java | 3 +-
.../runners/spark/translation/DoFnFunction.java | 41 +-
.../translation/DoFnRunnerWithMetrics.java | 91 +
.../spark/translation/EvaluationContext.java | 89 +-
.../translation/GroupCombineFunctions.java | 259 +-
.../spark/translation/MultiDoFnFunction.java | 39 +-
.../translation/SparkAbstractCombineFn.java | 12 +-
.../spark/translation/SparkContextFactory.java | 13 +-
.../spark/translation/SparkGlobalCombineFn.java | 13 +-
.../translation/SparkGroupAlsoByWindowFn.java | 214 -
...SparkGroupAlsoByWindowViaOutputBufferFn.java | 179 +
.../spark/translation/SparkKeyedCombineFn.java | 13 +-
.../spark/translation/SparkPCollectionView.java | 99 +
.../spark/translation/SparkProcessContext.java | 8 +-
.../spark/translation/SparkRuntimeContext.java | 84 +-
.../spark/translation/TransformEvaluator.java | 1 +
.../spark/translation/TransformTranslator.java | 467 +-
.../spark/translation/TranslationUtils.java | 93 +-
.../spark/translation/streaming/Checkpoint.java | 137 +
.../SparkRunnerStreamingContextFactory.java | 57 +-
.../streaming/StreamingTransformTranslator.java | 492 +-
.../translation/streaming/UnboundedDataset.java | 63 +-
.../runners/spark/util/BroadcastHelper.java | 127 -
.../spark/util/GlobalWatermarkHolder.java | 200 +
.../runners/spark/util/SideInputBroadcast.java | 77 +
.../spark/util/SparkSideInputReader.java | 8 +-
.../beam/runners/spark/ClearWatermarksRule.java | 37 +
.../beam/runners/spark/ForceStreamingTest.java | 60 +-
.../spark/GlobalWatermarkHolderTest.java | 151 +
.../apache/beam/runners/spark/PipelineRule.java | 109 +
.../runners/spark/ProvidedSparkContextTest.java | 72 +-
.../runners/spark/ReuseSparkContextRule.java | 46 +
.../runners/spark/SparkPipelineStateTest.java | 27 +-
.../runners/spark/SparkRunnerDebuggerTest.java | 180 +
.../spark/aggregators/ClearAggregatorsRule.java | 5 +-
.../metrics/sink/InMemoryMetrics.java | 10 +-
.../metrics/sink/NamedAggregatorsTest.java | 25 +-
.../coders/BeamSparkRunnerRegistratorTest.java | 57 -
.../runners/spark/coders/WritableCoderTest.java | 45 -
.../beam/runners/spark/io/AvroPipelineTest.java | 6 +-
.../beam/runners/spark/io/NumShardsTest.java | 6 +-
.../io/hadoop/HadoopFileFormatPipelineTest.java | 8 +-
.../spark/metrics/SparkBeamMetricTest.java | 60 +
.../spark/translation/StorageLevelTest.java | 8 +-
.../translation/streaming/CreateStreamTest.java | 376 ++
.../streaming/EmptyStreamAssertionTest.java | 87 -
.../streaming/FlattenStreamingTest.java | 103 -
.../streaming/KafkaStreamingTest.java | 215 -
.../ResumeFromCheckpointStreamingTest.java | 308 +-
.../streaming/SimpleStreamingWordCountTest.java | 84 -
.../streaming/TrackStreamingSourcesTest.java | 171 +
.../utils/KafkaWriteOnBatchCompleted.java | 105 -
.../streaming/utils/PAssertStreaming.java | 121 -
.../utils/SparkTestPipelineOptions.java | 42 -
.../SparkTestPipelineOptionsForStreaming.java | 37 -
.../spark/src/test/resources/log4j.properties | 30 +
.../spark/src/test/resources/metrics.properties | 10 +-
sdks/common/fn-api/pom.xml | 109 +
.../fn-api/src/main/proto/beam_fn_api.proto | 771 +++
.../org/apache/beam/fn/v1/standard_coders.yaml | 195 +
sdks/common/pom.xml | 39 +
sdks/common/runner-api/pom.xml | 89 +
.../src/main/proto/beam_runner_api.proto | 711 +++
sdks/java/build-tools/pom.xml | 2 +-
.../src/main/resources/beam/findbugs-filter.xml | 95 +-
sdks/java/core/pom.xml | 28 +-
.../main/java/org/apache/beam/sdk/Pipeline.java | 75 +-
.../beam/sdk/annotations/Experimental.java | 5 +-
.../org/apache/beam/sdk/coders/AtomicCoder.java | 2 +-
.../org/apache/beam/sdk/coders/AvroCoder.java | 30 +-
.../apache/beam/sdk/coders/BigDecimalCoder.java | 6 +-
.../beam/sdk/coders/BigEndianIntegerCoder.java | 7 +
.../beam/sdk/coders/BigEndianLongCoder.java | 7 +
.../apache/beam/sdk/coders/ByteArrayCoder.java | 7 +
.../org/apache/beam/sdk/coders/ByteCoder.java | 7 +
.../apache/beam/sdk/coders/ByteStringCoder.java | 8 +
.../java/org/apache/beam/sdk/coders/Coder.java | 7 +
.../apache/beam/sdk/coders/CollectionCoder.java | 12 +-
.../org/apache/beam/sdk/coders/CustomCoder.java | 18 +-
.../apache/beam/sdk/coders/DelegateCoder.java | 29 +-
.../org/apache/beam/sdk/coders/DoubleCoder.java | 7 +
.../apache/beam/sdk/coders/DurationCoder.java | 8 +
.../apache/beam/sdk/coders/InstantCoder.java | 7 +
.../apache/beam/sdk/coders/IterableCoder.java | 12 +-
.../beam/sdk/coders/IterableLikeCoder.java | 6 +-
.../org/apache/beam/sdk/coders/JAXBCoder.java | 48 +-
.../org/apache/beam/sdk/coders/KvCoder.java | 35 +-
.../beam/sdk/coders/LengthPrefixCoder.java | 145 +
.../org/apache/beam/sdk/coders/ListCoder.java | 7 +
.../org/apache/beam/sdk/coders/MapCoder.java | 62 +-
.../apache/beam/sdk/coders/NullableCoder.java | 6 +
.../beam/sdk/coders/SerializableCoder.java | 17 +-
.../org/apache/beam/sdk/coders/SetCoder.java | 12 +-
.../apache/beam/sdk/coders/StandardCoder.java | 42 +-
.../beam/sdk/coders/StringDelegateCoder.java | 16 +-
.../apache/beam/sdk/coders/StringUtf8Coder.java | 18 +-
.../beam/sdk/coders/TableRowJsonCoder.java | 7 +
.../beam/sdk/coders/TextualIntegerCoder.java | 8 +
.../org/apache/beam/sdk/coders/VarIntCoder.java | 10 +-
.../apache/beam/sdk/coders/VarLongCoder.java | 7 +
.../org/apache/beam/sdk/coders/VoidCoder.java | 7 +
.../beam/sdk/coders/protobuf/ProtoCoder.java | 8 +-
.../java/org/apache/beam/sdk/io/AvroIO.java | 4 +-
.../java/org/apache/beam/sdk/io/AvroSource.java | 5 -
.../sdk/io/BoundedReadFromUnboundedSource.java | 79 +-
.../org/apache/beam/sdk/io/BoundedSource.java | 8 -
.../apache/beam/sdk/io/CompressedSource.java | 40 +-
.../org/apache/beam/sdk/io/CountingSource.java | 5 -
.../org/apache/beam/sdk/io/FileBasedSink.java | 22 +
.../java/org/apache/beam/sdk/io/FileSystem.java | 115 +-
.../org/apache/beam/sdk/io/FileSystems.java | 88 +-
.../org/apache/beam/sdk/io/LocalFileSystem.java | 195 +-
.../org/apache/beam/sdk/io/LocalResourceId.java | 136 +
.../java/org/apache/beam/sdk/io/PubsubIO.java | 1142 ++---
.../apache/beam/sdk/io/PubsubUnboundedSink.java | 88 +-
.../beam/sdk/io/PubsubUnboundedSource.java | 104 +-
.../main/java/org/apache/beam/sdk/io/Read.java | 7 +-
.../java/org/apache/beam/sdk/io/TextIO.java | 401 +-
.../main/java/org/apache/beam/sdk/io/Write.java | 706 +--
.../java/org/apache/beam/sdk/io/XmlSource.java | 5 -
.../apache/beam/sdk/io/fs/CreateOptions.java | 60 +
.../org/apache/beam/sdk/io/fs/MatchResult.java | 125 +
.../apache/beam/sdk/io/fs/ResolveOptions.java | 41 +
.../org/apache/beam/sdk/io/fs/ResourceId.java | 85 +
.../org/apache/beam/sdk/io/fs/package-info.java | 22 +
.../beam/sdk/metrics/DistributionData.java | 3 +-
.../org/apache/beam/sdk/metrics/MetricKey.java | 3 +-
.../apache/beam/sdk/metrics/MetricUpdates.java | 3 +-
.../org/apache/beam/sdk/options/GcpOptions.java | 25 +-
.../org/apache/beam/sdk/options/GcsOptions.java | 4 +-
.../beam/sdk/options/PipelineOptions.java | 2 +-
.../sdk/options/PipelineOptionsFactory.java | 10 +-
.../apache/beam/sdk/options/ValueProvider.java | 6 +-
.../beam/sdk/runners/PTransformMatcher.java | 32 +
.../sdk/runners/PTransformOverrideFactory.java | 31 +
.../apache/beam/sdk/runners/PipelineRunner.java | 14 -
.../beam/sdk/runners/TransformHierarchy.java | 285 +-
.../apache/beam/sdk/testing/Annotations.java | 72 +
.../beam/sdk/testing/CoderProperties.java | 85 +-
.../testing/FlattenWithHeterogeneousCoders.java | 29 +
.../org/apache/beam/sdk/testing/PAssert.java | 259 +-
.../apache/beam/sdk/testing/RegexMatcher.java | 49 +
.../beam/sdk/testing/RunnableOnService.java | 14 +-
.../beam/sdk/testing/SourceTestUtils.java | 5 -
.../apache/beam/sdk/testing/TestPipeline.java | 111 +-
.../org/apache/beam/sdk/testing/TestStream.java | 18 +-
.../beam/sdk/testing/UsesAttemptedMetrics.java | 28 +
.../beam/sdk/testing/UsesCommittedMetrics.java | 28 +
.../apache/beam/sdk/testing/UsesMapState.java | 25 +
.../apache/beam/sdk/testing/UsesMetrics.java | 24 -
.../apache/beam/sdk/testing/UsesSetState.java | 25 +
.../apache/beam/sdk/testing/UsesTestStream.java | 24 +
.../sdk/testing/UsesUnboundedPCollections.java | 23 +
.../beam/sdk/testing/ValueInSingleWindow.java | 6 +-
.../apache/beam/sdk/transforms/Aggregator.java | 19 -
.../sdk/transforms/AggregatorRetriever.java | 13 +-
.../beam/sdk/transforms/AppliedPTransform.java | 31 +-
.../org/apache/beam/sdk/transforms/Combine.java | 204 +-
.../apache/beam/sdk/transforms/CombineFns.java | 14 +-
.../org/apache/beam/sdk/transforms/Count.java | 28 +-
.../org/apache/beam/sdk/transforms/Create.java | 88 +-
.../sdk/transforms/DelegatingAggregator.java | 2 +-
.../org/apache/beam/sdk/transforms/DoFn.java | 77 +-
.../beam/sdk/transforms/DoFnAdapters.java | 504 --
.../apache/beam/sdk/transforms/DoFnTester.java | 77 +-
.../org/apache/beam/sdk/transforms/Flatten.java | 15 +-
.../apache/beam/sdk/transforms/GroupByKey.java | 10 +-
.../org/apache/beam/sdk/transforms/Latest.java | 80 +-
.../org/apache/beam/sdk/transforms/Max.java | 124 +-
.../org/apache/beam/sdk/transforms/Mean.java | 27 +-
.../org/apache/beam/sdk/transforms/Min.java | 122 +-
.../org/apache/beam/sdk/transforms/OldDoFn.java | 758 ---
.../apache/beam/sdk/transforms/PTransform.java | 9 +-
.../org/apache/beam/sdk/transforms/ParDo.java | 55 +-
.../org/apache/beam/sdk/transforms/Regex.java | 589 ++-
.../org/apache/beam/sdk/transforms/Sample.java | 121 +-
.../beam/sdk/transforms/SimpleFunction.java | 44 +-
.../org/apache/beam/sdk/transforms/Sum.java | 57 +-
.../apache/beam/sdk/transforms/ToString.java | 181 +
.../org/apache/beam/sdk/transforms/Top.java | 27 +-
.../org/apache/beam/sdk/transforms/View.java | 10 +
.../sdk/transforms/display/DisplayData.java | 6 +-
.../beam/sdk/transforms/join/CoGbkResult.java | 35 +-
.../transforms/join/KeyedPCollectionTuple.java | 41 +-
.../reflect/ByteBuddyDoFnInvokerFactory.java | 16 +-
.../reflect/ByteBuddyOnTimerInvokerFactory.java | 4 +-
.../sdk/transforms/reflect/DoFnInvoker.java | 42 +-
.../sdk/transforms/reflect/DoFnInvokers.java | 142 +-
.../sdk/transforms/reflect/DoFnSignature.java | 71 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 82 +-
.../transforms/splittabledofn/OffsetRange.java | 71 +
.../splittabledofn/OffsetRangeTracker.java | 75 +
.../splittabledofn/RestrictionTracker.java | 2 +-
.../beam/sdk/transforms/windowing/AfterAll.java | 7 +
.../windowing/AfterDelayFromFirstElement.java | 251 -
.../sdk/transforms/windowing/AfterEach.java | 7 +
.../sdk/transforms/windowing/AfterFirst.java | 7 +
.../sdk/transforms/windowing/AfterPane.java | 10 -
.../windowing/AfterProcessingTime.java | 105 +-
.../AfterSynchronizedProcessingTime.java | 31 +-
.../sdk/transforms/windowing/BoundedWindow.java | 31 +
.../sdk/transforms/windowing/GlobalWindow.java | 6 +
.../sdk/transforms/windowing/GlobalWindows.java | 10 +
.../transforms/windowing/IntervalWindow.java | 19 +-
.../sdk/transforms/windowing/OutputTimeFns.java | 45 +
.../beam/sdk/transforms/windowing/PaneInfo.java | 2 -
.../windowing/TimestampTransform.java | 65 +
.../beam/sdk/transforms/windowing/Triggers.java | 320 ++
.../beam/sdk/transforms/windowing/Window.java | 96 +-
.../org/apache/beam/sdk/util/ApiSurface.java | 446 +-
.../beam/sdk/util/BaseExecutionContext.java | 174 -
.../org/apache/beam/sdk/util/CoderUtils.java | 30 +-
.../beam/sdk/util/CombineContextFactory.java | 18 -
.../org/apache/beam/sdk/util/DefaultBucket.java | 105 +
.../util/EmptyOnDeserializationThreadLocal.java | 39 +
.../apache/beam/sdk/util/ExecutionContext.java | 100 -
.../apache/beam/sdk/util/GcpProjectUtil.java | 2 +-
.../apache/beam/sdk/util/GcsPathValidator.java | 3 +-
.../java/org/apache/beam/sdk/util/GcsUtil.java | 334 +-
.../org/apache/beam/sdk/util/NameUtils.java | 167 +
.../org/apache/beam/sdk/util/PropertyNames.java | 1 +
.../org/apache/beam/sdk/util/PubsubClient.java | 28 +-
.../apache/beam/sdk/util/PubsubGrpcClient.java | 6 +-
.../apache/beam/sdk/util/PubsubJsonClient.java | 4 +-
.../apache/beam/sdk/util/PubsubTestClient.java | 6 +-
.../org/apache/beam/sdk/util/StringUtils.java | 100 -
.../java/org/apache/beam/sdk/util/Timer.java | 11 +
.../apache/beam/sdk/util/TimerInternals.java | 273 --
.../org/apache/beam/sdk/util/WindowedValue.java | 23 +-
.../beam/sdk/util/WindowingInternals.java | 82 -
.../beam/sdk/util/WindowingStrategies.java | 266 ++
.../apache/beam/sdk/util/WindowingStrategy.java | 53 +-
.../beam/sdk/util/common/ReflectHelpers.java | 16 +-
.../sdk/util/state/InMemoryStateInternals.java | 430 --
.../sdk/util/state/InMemoryTimerInternals.java | 275 --
.../apache/beam/sdk/util/state/MapState.java | 93 +
.../sdk/util/state/MergingStateAccessor.java | 40 -
.../beam/sdk/util/state/ReadableState.java | 4 +-
.../apache/beam/sdk/util/state/SetState.java | 71 +
.../beam/sdk/util/state/StateAccessor.java | 37 -
.../apache/beam/sdk/util/state/StateBinder.java | 6 +
.../beam/sdk/util/state/StateContexts.java | 63 -
.../beam/sdk/util/state/StateInternals.java | 57 -
.../sdk/util/state/StateInternalsFactory.java | 35 -
.../beam/sdk/util/state/StateMerging.java | 259 --
.../beam/sdk/util/state/StateNamespace.java | 56 -
.../sdk/util/state/StateNamespaceForTest.java | 65 -
.../beam/sdk/util/state/StateNamespaces.java | 278 --
.../apache/beam/sdk/util/state/StateSpecs.java | 155 +-
.../apache/beam/sdk/util/state/StateTable.java | 82 -
.../apache/beam/sdk/util/state/StateTag.java | 111 -
.../apache/beam/sdk/util/state/StateTags.java | 290 --
.../util/state/TestInMemoryStateInternals.java | 61 -
.../sdk/util/state/TimerInternalsFactory.java | 36 -
.../java/org/apache/beam/sdk/values/PBegin.java | 9 +-
.../apache/beam/sdk/values/PCollectionList.java | 62 +-
.../beam/sdk/values/PCollectionTuple.java | 33 +-
.../java/org/apache/beam/sdk/values/PDone.java | 4 +-
.../java/org/apache/beam/sdk/values/PInput.java | 13 +-
.../org/apache/beam/sdk/values/POutput.java | 24 +-
.../beam/sdk/values/POutputValueBase.java | 4 +-
.../java/org/apache/beam/sdk/values/PValue.java | 24 +-
.../org/apache/beam/sdk/values/PValueBase.java | 18 +-
.../apache/beam/sdk/values/TaggedPValue.java | 42 +
.../beam/sdk/values/TimestampedValue.java | 10 +-
.../apache/beam/sdk/values/TupleTagList.java | 6 +
.../org/apache/beam/sdk/values/TypedPValue.java | 78 +-
.../org/apache/beam/SdkCoreApiSurfaceTest.java | 62 +
.../sdk/AggregatorPipelineExtractorTest.java | 16 +-
.../java/org/apache/beam/sdk/PipelineTest.java | 37 +-
.../apache/beam/sdk/coders/AvroCoderTest.java | 18 +-
.../beam/sdk/coders/BigDecimalCoderTest.java | 46 +-
.../sdk/coders/BigEndianIntegerCoderTest.java | 9 +
.../beam/sdk/coders/BigEndianLongCoderTest.java | 9 +
.../beam/sdk/coders/ByteArrayCoderTest.java | 6 +
.../apache/beam/sdk/coders/ByteCoderTest.java | 9 +
.../beam/sdk/coders/ByteStringCoderTest.java | 8 +
.../beam/sdk/coders/CoderRegistryTest.java | 12 +-
.../org/apache/beam/sdk/coders/CoderTest.java | 8 +
.../beam/sdk/coders/CollectionCoderTest.java | 16 +
.../apache/beam/sdk/coders/CommonCoderTest.java | 351 ++
.../beam/sdk/coders/DefaultCoderTest.java | 4 +-
.../beam/sdk/coders/DelegateCoderTest.java | 35 +-
.../apache/beam/sdk/coders/DoubleCoderTest.java | 9 +
.../beam/sdk/coders/DurationCoderTest.java | 10 +
.../beam/sdk/coders/InstantCoderTest.java | 9 +
.../beam/sdk/coders/IterableCoderTest.java | 27 +-
.../apache/beam/sdk/coders/JAXBCoderTest.java | 26 +-
.../org/apache/beam/sdk/coders/KvCoderTest.java | 29 +
.../beam/sdk/coders/LengthPrefixCoderTest.java | 129 +
.../apache/beam/sdk/coders/ListCoderTest.java | 16 +-
.../apache/beam/sdk/coders/MapCoderTest.java | 21 +-
.../beam/sdk/coders/NullableCoderTest.java | 12 +
.../beam/sdk/coders/SerializableCoderTest.java | 16 +-
.../apache/beam/sdk/coders/SetCoderTest.java | 16 +
.../beam/sdk/coders/StandardCoderTest.java | 40 +
.../sdk/coders/StringDelegateCoderTest.java | 11 +
.../beam/sdk/coders/StringUtf8CoderTest.java | 9 +
.../beam/sdk/coders/TableRowJsonCoderTest.java | 9 +
.../sdk/coders/TextualIntegerCoderTest.java | 9 +
.../apache/beam/sdk/coders/VarIntCoderTest.java | 9 +
.../beam/sdk/coders/VarLongCoderTest.java | 9 +
.../apache/beam/sdk/coders/VoidCoderTest.java | 40 +
.../beam/sdk/io/AvroIOGeneratedClassTest.java | 285 --
.../java/org/apache/beam/sdk/io/AvroIOTest.java | 18 +-
.../apache/beam/sdk/io/AvroIOTransformTest.java | 324 ++
.../io/BoundedReadFromUnboundedSourceTest.java | 6 +-
.../beam/sdk/io/CompressedSourceTest.java | 89 +-
.../apache/beam/sdk/io/CountingInputTest.java | 12 +-
.../apache/beam/sdk/io/CountingSourceTest.java | 13 +-
.../apache/beam/sdk/io/FileBasedSinkTest.java | 17 +-
.../apache/beam/sdk/io/FileBasedSourceTest.java | 9 +-
.../org/apache/beam/sdk/io/FileSystemsTest.java | 61 +-
.../apache/beam/sdk/io/LocalFileSystemTest.java | 318 ++
.../apache/beam/sdk/io/LocalResourceIdTest.java | 226 +
.../beam/sdk/io/OffsetBasedSourceTest.java | 5 -
.../org/apache/beam/sdk/io/PubsubIOTest.java | 86 +-
.../beam/sdk/io/PubsubUnboundedSinkTest.java | 43 +-
.../beam/sdk/io/PubsubUnboundedSourceTest.java | 22 +-
.../java/org/apache/beam/sdk/io/ReadTest.java | 5 -
.../java/org/apache/beam/sdk/io/TextIOTest.java | 222 +-
.../java/org/apache/beam/sdk/io/WriteTest.java | 158 +-
.../org/apache/beam/sdk/io/XmlSourceTest.java | 10 +-
.../apache/beam/sdk/metrics/MetricMatchers.java | 192 +-
.../apache/beam/sdk/metrics/MetricsTest.java | 124 +-
.../apache/beam/sdk/options/GcpOptionsTest.java | 32 +-
.../sdk/options/PipelineOptionsFactoryTest.java | 6 +-
.../beam/sdk/options/PipelineOptionsTest.java | 3 +-
.../sdk/options/ProxyInvocationHandlerTest.java | 5 +-
.../beam/sdk/options/ValueProviderTest.java | 36 +-
.../sdk/options/ValueProviderUtilsTest.java | 2 +-
.../sdk/runners/TransformHierarchyTest.java | 320 +-
.../beam/sdk/runners/TransformTreeTest.java | 34 +-
.../beam/sdk/testing/GatherAllPanesTest.java | 7 +-
.../apache/beam/sdk/testing/PAssertTest.java | 118 +-
.../beam/sdk/testing/TestPipelineTest.java | 504 +-
.../apache/beam/sdk/testing/TestStreamTest.java | 45 +-
.../testing/ValueInSingleWindowCoderTest.java | 7 +
.../transforms/ApproximateQuantilesTest.java | 12 +-
.../sdk/transforms/ApproximateUniqueTest.java | 487 +-
.../beam/sdk/transforms/CombineFnsTest.java | 25 +-
.../apache/beam/sdk/transforms/CombineTest.java | 174 +-
.../apache/beam/sdk/transforms/CountTest.java | 15 +-
.../apache/beam/sdk/transforms/CreateTest.java | 119 +-
.../beam/sdk/transforms/DistinctTest.java | 12 +-
.../DoFnDelegatingAggregatorTest.java | 142 -
.../apache/beam/sdk/transforms/DoFnTest.java | 19 +-
.../beam/sdk/transforms/DoFnTesterTest.java | 12 +-
.../apache/beam/sdk/transforms/FilterTest.java | 18 +-
.../sdk/transforms/FlatMapElementsTest.java | 10 +-
.../apache/beam/sdk/transforms/FlattenTest.java | 74 +-
.../beam/sdk/transforms/GroupByKeyTest.java | 32 +-
.../apache/beam/sdk/transforms/KeysTest.java | 9 +-
.../apache/beam/sdk/transforms/KvSwapTest.java | 22 +-
.../apache/beam/sdk/transforms/LatestTest.java | 23 +-
.../beam/sdk/transforms/MapElementsTest.java | 14 +-
.../org/apache/beam/sdk/transforms/MaxTest.java | 20 +-
.../apache/beam/sdk/transforms/MeanTest.java | 7 +-
.../org/apache/beam/sdk/transforms/MinTest.java | 21 +-
.../apache/beam/sdk/transforms/NoOpOldDoFn.java | 71 -
.../beam/sdk/transforms/OldDoFnContextTest.java | 69 -
.../apache/beam/sdk/transforms/OldDoFnTest.java | 188 -
.../beam/sdk/transforms/ParDoLifecycleTest.java | 17 +-
.../apache/beam/sdk/transforms/ParDoTest.java | 704 ++-
.../beam/sdk/transforms/PartitionTest.java | 8 +-
.../apache/beam/sdk/transforms/RegexTest.java | 148 +-
.../apache/beam/sdk/transforms/SampleTest.java | 419 +-
.../beam/sdk/transforms/SimpleFunctionTest.java | 43 +
.../beam/sdk/transforms/SimpleStatsFnsTest.java | 36 +-
.../beam/sdk/transforms/SplittableDoFnTest.java | 83 +-
.../org/apache/beam/sdk/transforms/SumTest.java | 24 +-
.../beam/sdk/transforms/ToStringTest.java | 125 +
.../org/apache/beam/sdk/transforms/TopTest.java | 33 +-
.../apache/beam/sdk/transforms/ValuesTest.java | 7 +-
.../apache/beam/sdk/transforms/ViewTest.java | 108 +-
.../beam/sdk/transforms/WithKeysTest.java | 8 +-
.../beam/sdk/transforms/WithTimestampsTest.java | 9 +-
.../display/DisplayDataEvaluator.java | 8 +-
.../sdk/transforms/display/DisplayDataTest.java | 15 +
.../transforms/join/CoGbkResultCoderTest.java | 10 +-
.../sdk/transforms/join/CoGroupByKeyTest.java | 11 +-
.../sdk/transforms/join/UnionCoderTest.java | 17 +-
.../transforms/reflect/DoFnInvokersTest.java | 70 +-
.../DoFnSignaturesProcessElementTest.java | 40 +-
.../DoFnSignaturesSplittableDoFnTest.java | 3 +-
.../transforms/reflect/DoFnSignaturesTest.java | 6 +-
.../splittabledofn/OffsetRangeTrackerTest.java | 111 +
.../windowing/AfterProcessingTimeTest.java | 2 +-
.../AfterSynchronizedProcessingTimeTest.java | 2 +-
.../transforms/windowing/GlobalWindowTest.java | 64 +
.../transforms/windowing/OutputTimeFnsTest.java | 51 +
.../sdk/transforms/windowing/TriggersTest.java | 100 +
.../sdk/transforms/windowing/WindowTest.java | 204 +-
.../sdk/transforms/windowing/WindowingTest.java | 15 +-
.../apache/beam/sdk/util/ApiSurfaceTest.java | 152 +-
.../apache/beam/sdk/util/CombineFnUtilTest.java | 8 +-
.../apache/beam/sdk/util/DefaultBucketTest.java | 112 +
.../beam/sdk/util/FileIOChannelFactoryTest.java | 13 +-
.../beam/sdk/util/GcsPathValidatorTest.java | 17 +-
.../org/apache/beam/sdk/util/GcsUtilTest.java | 86 +-
.../org/apache/beam/sdk/util/NameUtilsTest.java | 177 +
.../beam/sdk/util/PubsubGrpcClientTest.java | 8 +-
.../beam/sdk/util/PubsubJsonClientTest.java | 3 +-
.../beam/sdk/util/PubsubTestClientTest.java | 4 +-
.../org/apache/beam/sdk/util/ReshuffleTest.java | 11 +-
.../beam/sdk/util/SerializableUtilsTest.java | 4 +-
.../apache/beam/sdk/util/StringUtilsTest.java | 100 -
.../beam/sdk/util/TimerInternalsTest.java | 101 -
.../beam/sdk/util/ValueWithRecordIdTest.java | 34 +
.../apache/beam/sdk/util/WindowedValueTest.java | 23 +
.../beam/sdk/util/WindowingStrategiesTest.java | 91 +
.../util/state/InMemoryStateInternalsTest.java | 348 --
.../util/state/InMemoryTimerInternalsTest.java | 153 -
.../sdk/util/state/StateNamespacesTest.java | 130 -
.../beam/sdk/util/state/StateTagTest.java | 173 -
.../beam/sdk/values/PCollectionListTest.java | 117 +
.../beam/sdk/values/PCollectionTupleTest.java | 80 +-
.../org/apache/beam/sdk/values/PDoneTest.java | 9 +-
.../beam/sdk/values/TimestampedValueTest.java | 19 +-
.../apache/beam/sdk/values/TypedPValueTest.java | 17 +-
sdks/java/extensions/jackson/pom.xml | 125 +
.../beam/sdk/extensions/jackson/AsJsons.java | 76 +
.../beam/sdk/extensions/jackson/ParseJsons.java | 75 +
.../sdk/extensions/jackson/package-info.java | 22 +
.../jackson/JacksonTransformsTest.java | 242 +
sdks/java/extensions/join-library/README.md | 10 -
sdks/java/extensions/join-library/pom.xml | 3 +-
.../extensions/joinlibrary/InnerJoinTest.java | 23 +-
.../joinlibrary/OuterLeftJoinTest.java | 31 +-
.../joinlibrary/OuterRightJoinTest.java | 31 +-
sdks/java/extensions/pom.xml | 3 +-
sdks/java/extensions/sorter/pom.xml | 5 +-
.../sorter/BufferedExternalSorter.java | 23 +-
.../sorter/BufferedExternalSorterTest.java | 46 +-
.../sdk/extensions/sorter/SortValuesTest.java | 9 +-
sdks/java/harness/pom.xml | 177 +
.../org/apache/beam/fn/harness/FnHarness.java | 131 +
.../harness/channel/ManagedChannelFactory.java | 80 +
.../harness/channel/SocketAddressFactory.java | 64 +
.../beam/fn/harness/channel/package-info.java | 22 +
.../fn/harness/control/BeamFnControlClient.java | 166 +
.../harness/control/ProcessBundleHandler.java | 334 ++
.../fn/harness/control/RegisterHandler.java | 92 +
.../beam/fn/harness/control/package-info.java | 22 +
.../BeamFnDataBufferingOutboundObserver.java | 135 +
.../beam/fn/harness/data/BeamFnDataClient.java | 64 +
.../fn/harness/data/BeamFnDataGrpcClient.java | 122 +
.../harness/data/BeamFnDataGrpcMultiplexer.java | 141 +
.../harness/data/BeamFnDataInboundObserver.java | 81 +
.../beam/fn/harness/data/package-info.java | 22 +
.../fn/harness/fake/FakeAggregatorFactory.java | 52 +
.../beam/fn/harness/fake/FakeStepContext.java | 70 +
.../beam/fn/harness/fake/package-info.java | 22 +
.../harness/fn/CloseableThrowingConsumer.java | 23 +
.../beam/fn/harness/fn/ThrowingBiFunction.java | 32 +
.../beam/fn/harness/fn/ThrowingConsumer.java | 32 +
.../beam/fn/harness/fn/ThrowingFunction.java | 32 +
.../beam/fn/harness/fn/ThrowingRunnable.java | 30 +
.../apache/beam/fn/harness/fn/package-info.java | 22 +
.../fn/harness/logging/BeamFnLoggingClient.java | 310 ++
.../beam/fn/harness/logging/package-info.java | 22 +
.../apache/beam/fn/harness/package-info.java | 22 +
.../beam/fn/harness/stream/AdvancingPhaser.java | 36 +
.../harness/stream/BufferingStreamObserver.java | 166 +
.../fn/harness/stream/DirectStreamObserver.java | 71 +
.../ForwardingClientResponseObserver.java | 63 +
.../harness/stream/StreamObserverFactory.java | 91 +
.../beam/fn/harness/stream/package-info.java | 22 +
.../beam/runners/core/BeamFnDataReadRunner.java | 104 +
.../runners/core/BeamFnDataWriteRunner.java | 87 +
.../beam/runners/core/BoundedSourceRunner.java | 105 +
.../apache/beam/runners/core/package-info.java | 22 +
.../apache/beam/fn/harness/FnHarnessTest.java | 130 +
.../channel/ManagedChannelFactoryTest.java | 74 +
.../channel/SocketAddressFactoryTest.java | 56 +
.../control/BeamFnControlClientTest.java | 182 +
.../control/ProcessBundleHandlerTest.java | 675 +++
.../fn/harness/control/RegisterHandlerTest.java | 80 +
...BeamFnDataBufferingOutboundObserverTest.java | 147 +
.../harness/data/BeamFnDataGrpcClientTest.java | 318 ++
.../data/BeamFnDataGrpcMultiplexerTest.java | 98 +
.../data/BeamFnDataInboundObserverTest.java | 116 +
.../logging/BeamFnLoggingClientTest.java | 169 +
.../fn/harness/stream/AdvancingPhaserTest.java | 48 +
.../stream/BufferingStreamObserverTest.java | 146 +
.../stream/DirectStreamObserverTest.java | 139 +
.../ForwardingClientResponseObserverTest.java | 60 +
.../stream/StreamObserverFactoryTest.java | 84 +
.../beam/fn/harness/test/TestExecutors.java | 85 +
.../beam/fn/harness/test/TestExecutorsTest.java | 160 +
.../beam/fn/harness/test/TestStreams.java | 162 +
.../beam/fn/harness/test/TestStreamsTest.java | 84 +
.../runners/core/BeamFnDataReadRunnerTest.java | 187 +
.../runners/core/BeamFnDataWriteRunnerTest.java | 155 +
.../runners/core/BoundedSourceRunnerTest.java | 113 +
sdks/java/io/elasticsearch/pom.xml | 149 +
.../sdk/io/elasticsearch/ElasticsearchIO.java | 815 ++++
.../beam/sdk/io/elasticsearch/package-info.java | 20 +
.../src/test/contrib/create_elk_container.sh | 24 +
.../elasticsearch/ElasticSearchIOTestUtils.java | 129 +
.../sdk/io/elasticsearch/ElasticsearchIOIT.java | 154 +
.../io/elasticsearch/ElasticsearchIOTest.java | 358 ++
.../elasticsearch/ElasticsearchTestDataSet.java | 109 +
.../elasticsearch/ElasticsearchTestOptions.java | 46 +
sdks/java/io/google-cloud-platform/pom.xml | 16 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 450 +-
.../sdk/io/gcp/bigquery/BigQueryServices.java | 23 +-
.../io/gcp/bigquery/BigQueryServicesImpl.java | 118 +-
.../gcp/bigquery/BigQueryTableRowIterator.java | 82 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 8 -
.../io/gcp/bigtable/BigtableTestOptions.java | 37 -
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 149 +-
.../beam/sdk/io/gcp/storage/GcsFileSystem.java | 210 +-
.../io/gcp/storage/GcsFileSystemRegistrar.java | 9 +-
.../beam/sdk/io/gcp/storage/GcsResourceId.java | 116 +
.../apache/beam/sdk/io/gcp/ApiSurfaceTest.java | 134 -
.../beam/sdk/io/gcp/GcpApiSurfaceTest.java | 79 +
.../sdk/io/gcp/bigquery/BigQueryIOTest.java | 335 +-
.../gcp/bigquery/BigQueryServicesImplTest.java | 141 +
.../bigquery/BigQueryTableRowIteratorTest.java | 51 +-
.../sdk/io/gcp/bigquery/BigQueryUtilTest.java | 3 +-
.../sdk/io/gcp/bigtable/BigtableIOTest.java | 24 +-
.../io/gcp/bigtable/BigtableTestOptions.java | 37 +
.../sdk/io/gcp/datastore/DatastoreV1Test.java | 15 +-
.../sdk/io/gcp/datastore/SplitQueryFnIT.java | 2 +-
.../sdk/io/gcp/storage/GcsFileSystemTest.java | 274 ++
.../sdk/io/gcp/storage/GcsResourceIdTest.java | 147 +
sdks/java/io/hadoop-common/pom.xml | 86 +
.../io/hadoop/SerializableConfiguration.java | 96 +
.../beam/sdk/io/hadoop/WritableCoder.java | 116 +
.../apache/beam/sdk/io/hadoop/package-info.java | 22 +
.../hadoop/SerializableConfigurationTest.java | 75 +
.../beam/sdk/io/hadoop/WritableCoderTest.java | 45 +
sdks/java/io/hadoop-input-format/README.md | 167 +
sdks/java/io/hadoop-input-format/pom.xml | 136 +
.../hadoop/inputformat/HadoopInputFormatIO.java | 941 ++++
.../sdk/io/hadoop/inputformat/package-info.java | 23 +
.../ConfigurableEmployeeInputFormat.java | 131 +
.../sdk/io/hadoop/inputformat/Employee.java | 85 +
.../hadoop/inputformat/EmployeeInputFormat.java | 172 +
.../inputformat/HadoopInputFormatIOTest.java | 844 ++++
.../ReuseObjectsEmployeeInputFormat.java | 176 +
.../hadoop/inputformat/TestEmployeeDataSet.java | 76 +
sdks/java/io/hbase/pom.xml | 233 +
.../org/apache/beam/sdk/io/hbase/HBaseIO.java | 693 +++
.../beam/sdk/io/hbase/HBaseMutationCoder.java | 71 +
.../beam/sdk/io/hbase/HBaseResultCoder.java | 55 +
.../beam/sdk/io/hbase/SerializableScan.java | 55 +
.../apache/beam/sdk/io/hbase/package-info.java | 24 +
.../apache/beam/sdk/io/hbase/HBaseIOTest.java | 430 ++
.../sdk/io/hbase/HBaseMutationCoderTest.java | 52 +
.../beam/sdk/io/hbase/HBaseResultCoderTest.java | 41 +
.../beam/sdk/io/hbase/SerializableScanTest.java | 56 +
sdks/java/io/hdfs/pom.xml | 64 +-
.../beam/sdk/io/hdfs/AvroHDFSFileSource.java | 142 -
.../beam/sdk/io/hdfs/AvroWrapperCoder.java | 114 -
.../apache/beam/sdk/io/hdfs/HDFSFileSink.java | 301 +-
.../apache/beam/sdk/io/hdfs/HDFSFileSource.java | 528 ++-
.../beam/sdk/io/hdfs/HadoopFileSystem.java | 44 +-
.../beam/sdk/io/hdfs/HadoopResourceId.java | 42 +
.../org/apache/beam/sdk/io/hdfs/UGIHelper.java | 38 +
.../apache/beam/sdk/io/hdfs/WritableCoder.java | 116 -
.../SimpleAuthAvroHDFSFileSource.java | 82 -
.../hdfs/simpleauth/SimpleAuthHDFSFileSink.java | 131 -
.../simpleauth/SimpleAuthHDFSFileSource.java | 117 -
.../sdk/io/hdfs/simpleauth/package-info.java | 22 -
.../beam/sdk/io/hdfs/AvroWrapperCoderTest.java | 51 -
.../beam/sdk/io/hdfs/HDFSFileSinkTest.java | 173 +
.../beam/sdk/io/hdfs/HDFSFileSourceTest.java | 60 +-
.../beam/sdk/io/hdfs/WritableCoderTest.java | 45 -
sdks/java/io/jdbc/pom.xml | 125 +-
sdks/java/io/jdbc/src/test/README.md | 32 +
.../org/apache/beam/sdk/io/jdbc/JdbcIOIT.java | 178 +
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 120 +-
.../beam/sdk/io/jdbc/JdbcTestDataSet.java | 128 +
.../beam/sdk/io/jdbc/PostgresTestOptions.java | 60 +
.../kubernetes/postgres-pod-no-vol.yml | 32 +
.../kubernetes/postgres-service-public.yml | 28 +
.../jdbc/src/test/resources/kubernetes/setup.sh | 20 +
.../src/test/resources/kubernetes/teardown.sh | 20 +
sdks/java/io/jms/pom.xml | 22 +-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 103 +-
.../org/apache/beam/sdk/io/jms/JmsIOTest.java | 137 +-
sdks/java/io/kafka/pom.xml | 65 +-
.../apache/beam/sdk/io/kafka/ConsumerSpEL.java | 60 +
.../beam/sdk/io/kafka/KafkaCheckpointMark.java | 16 +
.../org/apache/beam/sdk/io/kafka/KafkaIO.java | 683 ++-
.../beam/sdk/io/kafka/KafkaRecordCoder.java | 4 +-
.../apache/beam/sdk/io/kafka/KafkaIOTest.java | 109 +-
.../beam/sdk/io/kafka/KafkaRecordCoderTest.java | 34 +
sdks/java/io/kinesis/pom.xml | 12 +-
.../beam/sdk/io/kinesis/KinesisRecordCoder.java | 4 +-
.../beam/sdk/io/kinesis/package-info.java | 2 +-
.../sdk/io/kinesis/KinesisMockReadTest.java | 7 +-
.../beam/sdk/io/kinesis/KinesisReaderIT.java | 6 +-
sdks/java/io/mongodb/pom.xml | 20 +-
.../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 5 -
.../apache/beam/sdk/io/mongodb/MongoDbIO.java | 5 -
.../sdk/io/mongodb/MongoDBGridFSIOTest.java | 9 +-
.../beam/sdk/io/mongodb/MongoDbIOTest.java | 7 +-
sdks/java/io/mqtt/pom.xml | 122 +
.../org/apache/beam/sdk/io/mqtt/MqttIO.java | 592 +++
.../apache/beam/sdk/io/mqtt/package-info.java | 22 +
.../org/apache/beam/sdk/io/mqtt/MqttIOTest.java | 257 +
sdks/java/io/pom.xml | 76 +-
sdks/java/java8tests/pom.xml | 3 +-
.../beam/sdk/transforms/CombineJava8Test.java | 8 +-
.../beam/sdk/transforms/DistinctJava8Test.java | 5 +-
.../beam/sdk/transforms/FilterJava8Test.java | 9 +-
.../transforms/FlatMapElementsJava8Test.java | 7 +-
.../sdk/transforms/MapElementsJava8Test.java | 33 +-
.../beam/sdk/transforms/PartitionJava8Test.java | 7 +-
.../sdk/transforms/SimpleFunctionJava8Test.java | 69 +
.../beam/sdk/transforms/WithKeysJava8Test.java | 6 +-
.../sdk/transforms/WithTimestampsJava8Test.java | 14 +-
sdks/java/javadoc/ant.xml | 96 +
sdks/java/javadoc/pom.xml | 295 ++
.../maven-archetypes/examples-java8/pom.xml | 2 +-
.../main/resources/archetype-resources/pom.xml | 78 +-
sdks/java/maven-archetypes/examples/pom.xml | 2 +-
.../main/resources/archetype-resources/pom.xml | 96 +-
sdks/java/maven-archetypes/pom.xml | 2 +-
sdks/java/maven-archetypes/starter/pom.xml | 2 +-
.../main/resources/archetype-resources/pom.xml | 4 +-
.../resources/projects/basic/reference/pom.xml | 4 +-
sdks/java/pom.xml | 6 +-
sdks/pom.xml | 17 +-
sdks/python/.pylintrc | 164 +
sdks/python/MANIFEST.in | 21 +
sdks/python/README.md | 298 ++
sdks/python/apache_beam/__init__.py | 82 +
sdks/python/apache_beam/coders/__init__.py | 19 +
sdks/python/apache_beam/coders/coder_impl.pxd | 143 +
sdks/python/apache_beam/coders/coder_impl.py | 734 +++
sdks/python/apache_beam/coders/coders.py | 835 ++++
sdks/python/apache_beam/coders/coders_test.py | 115 +
.../apache_beam/coders/coders_test_common.py | 389 ++
.../apache_beam/coders/fast_coders_test.py | 37 +
sdks/python/apache_beam/coders/observable.py | 38 +
.../apache_beam/coders/observable_test.py | 57 +
.../coders/proto2_coder_test_messages_pb2.py | 318 ++
.../apache_beam/coders/slow_coders_test.py | 45 +
sdks/python/apache_beam/coders/slow_stream.py | 163 +
.../apache_beam/coders/standard_coders_test.py | 156 +
sdks/python/apache_beam/coders/stream.pxd | 66 +
sdks/python/apache_beam/coders/stream.pyx | 226 +
sdks/python/apache_beam/coders/stream_test.py | 180 +
sdks/python/apache_beam/coders/typecoders.py | 182 +
.../apache_beam/coders/typecoders_test.py | 124 +
sdks/python/apache_beam/error.py | 42 +
sdks/python/apache_beam/examples/__init__.py | 16 +
.../apache_beam/examples/complete/__init__.py | 16 +
.../examples/complete/autocomplete.py | 90 +
.../examples/complete/autocomplete_test.py | 52 +
.../examples/complete/estimate_pi.py | 128 +
.../examples/complete/estimate_pi_test.py | 52 +
.../examples/complete/juliaset/__init__.py | 16 +
.../complete/juliaset/juliaset/__init__.py | 16 +
.../complete/juliaset/juliaset/juliaset.py | 124 +
.../complete/juliaset/juliaset/juliaset_test.py | 86 +
.../examples/complete/juliaset/juliaset_main.py | 58 +
.../examples/complete/juliaset/setup.py | 116 +
.../apache_beam/examples/complete/tfidf.py | 208 +
.../apache_beam/examples/complete/tfidf_test.py | 91 +
.../examples/complete/top_wikipedia_sessions.py | 182 +
.../complete/top_wikipedia_sessions_test.py | 62 +
.../apache_beam/examples/cookbook/__init__.py | 16 +
.../examples/cookbook/bigquery_schema.py | 129 +
.../examples/cookbook/bigquery_side_input.py | 121 +
.../cookbook/bigquery_side_input_test.py | 54 +
.../examples/cookbook/bigquery_tornadoes.py | 99 +
.../cookbook/bigquery_tornadoes_it_test.py | 62 +
.../cookbook/bigquery_tornadoes_test.py | 45 +
.../apache_beam/examples/cookbook/bigshuffle.py | 94 +
.../examples/cookbook/bigshuffle_test.py | 63 +
.../apache_beam/examples/cookbook/coders.py | 101 +
.../examples/cookbook/coders_test.py | 49 +
.../examples/cookbook/combiners_test.py | 74 +
.../examples/cookbook/custom_ptransform.py | 134 +
.../examples/cookbook/custom_ptransform_test.py | 53 +
.../examples/cookbook/datastore_wordcount.py | 261 ++
.../apache_beam/examples/cookbook/filters.py | 107 +
.../examples/cookbook/filters_test.py | 69 +
.../examples/cookbook/group_with_coder.py | 122 +
.../examples/cookbook/group_with_coder_test.py | 89 +
.../examples/cookbook/mergecontacts.py | 133 +
.../examples/cookbook/mergecontacts_test.py | 125 +
.../examples/cookbook/multiple_output_pardo.py | 184 +
.../cookbook/multiple_output_pardo_test.py | 72 +
.../apache_beam/examples/snippets/__init__.py | 16 +
.../apache_beam/examples/snippets/snippets.py | 1158 +++++
.../examples/snippets/snippets_test.py | 904 ++++
.../apache_beam/examples/streaming_wordcap.py | 64 +
.../apache_beam/examples/streaming_wordcount.py | 74 +
sdks/python/apache_beam/examples/wordcount.py | 116 +
.../apache_beam/examples/wordcount_debugging.py | 163 +
.../examples/wordcount_debugging_test.py | 59 +
.../apache_beam/examples/wordcount_it_test.py | 59 +
.../apache_beam/examples/wordcount_minimal.py | 121 +
.../examples/wordcount_minimal_test.py | 59 +
.../apache_beam/examples/wordcount_test.py | 58 +
sdks/python/apache_beam/internal/__init__.py | 16 +
.../python/apache_beam/internal/gcp/__init__.py | 16 +
sdks/python/apache_beam/internal/gcp/auth.py | 185 +
.../apache_beam/internal/gcp/auth_test.py | 44 +
.../apache_beam/internal/gcp/json_value.py | 147 +
.../apache_beam/internal/gcp/json_value_test.py | 93 +
sdks/python/apache_beam/internal/module_test.py | 62 +
sdks/python/apache_beam/internal/pickler.py | 230 +
.../python/apache_beam/internal/pickler_test.py | 84 +
sdks/python/apache_beam/internal/util.py | 127 +
sdks/python/apache_beam/internal/util_test.py | 61 +
sdks/python/apache_beam/io/__init__.py | 38 +
sdks/python/apache_beam/io/avroio.py | 372 ++
sdks/python/apache_beam/io/avroio_test.py | 381 ++
sdks/python/apache_beam/io/concat_source.py | 263 ++
.../python/apache_beam/io/concat_source_test.py | 231 +
sdks/python/apache_beam/io/filebasedsource.py | 329 ++
.../apache_beam/io/filebasedsource_test.py | 708 +++
sdks/python/apache_beam/io/fileio.py | 746 +++
sdks/python/apache_beam/io/fileio_test.py | 352 ++
sdks/python/apache_beam/io/gcp/__init__.py | 16 +
sdks/python/apache_beam/io/gcp/bigquery.py | 1081 +++++
sdks/python/apache_beam/io/gcp/bigquery_test.py | 828 ++++
.../apache_beam/io/gcp/datastore/__init__.py | 16 +
.../apache_beam/io/gcp/datastore/v1/__init__.py | 16 +
.../io/gcp/datastore/v1/datastoreio.py | 397 ++
.../io/gcp/datastore/v1/datastoreio_test.py | 245 +
.../io/gcp/datastore/v1/fake_datastore.py | 98 +
.../apache_beam/io/gcp/datastore/v1/helper.py | 274 ++
.../io/gcp/datastore/v1/helper_test.py | 265 ++
.../io/gcp/datastore/v1/query_splitter.py | 275 ++
.../io/gcp/datastore/v1/query_splitter_test.py | 208 +
sdks/python/apache_beam/io/gcp/gcsio.py | 871 ++++
sdks/python/apache_beam/io/gcp/gcsio_test.py | 796 ++++
.../apache_beam/io/gcp/internal/__init__.py | 16 +
.../io/gcp/internal/clients/__init__.py | 16 +
.../gcp/internal/clients/bigquery/__init__.py | 33 +
.../clients/bigquery/bigquery_v2_client.py | 660 +++
.../clients/bigquery/bigquery_v2_messages.py | 1910 ++++++++
.../io/gcp/internal/clients/storage/__init__.py | 33 +
.../clients/storage/storage_v1_client.py | 1039 +++++
.../clients/storage/storage_v1_messages.py | 1920 ++++++++
sdks/python/apache_beam/io/gcp/pubsub.py | 91 +
sdks/python/apache_beam/io/gcp/pubsub_test.py | 63 +
.../python/apache_beam/io/gcp/tests/__init__.py | 16 +
.../io/gcp/tests/bigquery_matcher.py | 108 +
.../io/gcp/tests/bigquery_matcher_test.py | 108 +
sdks/python/apache_beam/io/iobase.py | 987 ++++
sdks/python/apache_beam/io/range_trackers.py | 532 +++
.../apache_beam/io/range_trackers_test.py | 590 +++
sdks/python/apache_beam/io/source_test_utils.py | 642 +++
.../apache_beam/io/source_test_utils_test.py | 122 +
sdks/python/apache_beam/io/sources_test.py | 111 +
sdks/python/apache_beam/io/textio.py | 448 ++
sdks/python/apache_beam/io/textio_test.py | 718 +++
sdks/python/apache_beam/io/tfrecordio.py | 271 ++
sdks/python/apache_beam/io/tfrecordio_test.py | 389 ++
sdks/python/apache_beam/metrics/__init__.py | 17 +
sdks/python/apache_beam/metrics/cells.py | 315 ++
sdks/python/apache_beam/metrics/cells_test.py | 143 +
sdks/python/apache_beam/metrics/execution.pxd | 31 +
sdks/python/apache_beam/metrics/execution.py | 229 +
.../apache_beam/metrics/execution_test.py | 131 +
sdks/python/apache_beam/metrics/metric.py | 202 +
sdks/python/apache_beam/metrics/metric_test.py | 128 +
sdks/python/apache_beam/metrics/metricbase.py | 82 +
sdks/python/apache_beam/pipeline.py | 442 ++
sdks/python/apache_beam/pipeline_test.py | 444 ++
sdks/python/apache_beam/pvalue.py | 468 ++
sdks/python/apache_beam/pvalue_test.py | 68 +
sdks/python/apache_beam/runners/__init__.py | 30 +
sdks/python/apache_beam/runners/api/__init__.py | 16 +
.../runners/api/beam_runner_api_pb2.py | 2772 +++++++++++
sdks/python/apache_beam/runners/common.pxd | 77 +
sdks/python/apache_beam/runners/common.py | 436 ++
.../apache_beam/runners/dataflow/__init__.py | 16 +
.../runners/dataflow/dataflow_metrics.py | 111 +
.../runners/dataflow/dataflow_metrics_test.py | 148 +
.../runners/dataflow/dataflow_runner.py | 729 +++
.../runners/dataflow/dataflow_runner_test.py | 181 +
.../runners/dataflow/internal/__init__.py | 16 +
.../runners/dataflow/internal/apiclient.py | 739 +++
.../runners/dataflow/internal/apiclient_test.py | 96 +
.../dataflow/internal/clients/__init__.py | 16 +
.../internal/clients/dataflow/__init__.py | 33 +
.../clients/dataflow/dataflow_v1b3_client.py | 694 +++
.../clients/dataflow/dataflow_v1b3_messages.py | 4392 ++++++++++++++++++
.../clients/dataflow/message_matchers.py | 124 +
.../clients/dataflow/message_matchers_test.py | 77 +
.../runners/dataflow/internal/dependency.py | 522 +++
.../dataflow/internal/dependency_test.py | 425 ++
.../runners/dataflow/internal/names.py | 82 +
.../runners/dataflow/native_io/__init__.py | 16 +
.../runners/dataflow/native_io/iobase.py | 318 ++
.../runners/dataflow/template_runner_test.py | 97 +
.../runners/dataflow/test_dataflow_runner.py | 40 +
.../apache_beam/runners/direct/__init__.py | 19 +
.../runners/direct/bundle_factory.py | 201 +
sdks/python/apache_beam/runners/direct/clock.py | 50 +
.../consumer_tracking_pipeline_visitor.py | 59 +
.../consumer_tracking_pipeline_visitor_test.py | 127 +
.../runners/direct/direct_metrics.py | 112 +
.../runners/direct/direct_metrics_test.py | 211 +
.../apache_beam/runners/direct/direct_runner.py | 173 +
.../runners/direct/evaluation_context.py | 283 ++
.../apache_beam/runners/direct/executor.py | 578 +++
.../runners/direct/helper_transforms.py | 99 +
.../runners/direct/transform_evaluator.py | 558 +++
.../runners/direct/transform_result.py | 64 +
.../runners/direct/watermark_manager.py | 224 +
.../apache_beam/runners/pipeline_context.py | 88 +
.../runners/pipeline_context_test.py | 49 +
sdks/python/apache_beam/runners/runner.py | 368 ++
sdks/python/apache_beam/runners/runner_test.py | 123 +
.../python/apache_beam/runners/test/__init__.py | 30 +
sdks/python/apache_beam/test_pipeline.py | 163 +
sdks/python/apache_beam/test_pipeline_test.py | 112 +
sdks/python/apache_beam/tests/__init__.py | 16 +
sdks/python/apache_beam/tests/data/README.md | 20 +
.../apache_beam/tests/data/privatekey.p12 | Bin 0 -> 2452 bytes
.../apache_beam/tests/data/standard_coders.yaml | 196 +
.../apache_beam/tests/pipeline_verifiers.py | 119 +
.../tests/pipeline_verifiers_test.py | 123 +
sdks/python/apache_beam/tests/test_utils.py | 69 +
sdks/python/apache_beam/transforms/__init__.py | 25 +
sdks/python/apache_beam/transforms/combiners.py | 595 +++
.../apache_beam/transforms/combiners_test.py | 324 ++
sdks/python/apache_beam/transforms/core.py | 1389 ++++++
.../apache_beam/transforms/cy_combiners.pxd | 92 +
.../apache_beam/transforms/cy_combiners.py | 306 ++
sdks/python/apache_beam/transforms/display.py | 331 ++
.../apache_beam/transforms/display_test.py | 216 +
.../python/apache_beam/transforms/ptransform.py | 671 +++
.../apache_beam/transforms/ptransform_test.py | 1941 ++++++++
.../python/apache_beam/transforms/sideinputs.py | 214 +
.../apache_beam/transforms/sideinputs_test.py | 337 ++
sdks/python/apache_beam/transforms/timeutil.py | 133 +
sdks/python/apache_beam/transforms/trigger.py | 1109 +++++
.../apache_beam/transforms/trigger_test.py | 601 +++
.../transforms/trigger_transcripts.yaml | 224 +
sdks/python/apache_beam/transforms/util.py | 235 +
sdks/python/apache_beam/transforms/window.py | 475 ++
.../apache_beam/transforms/window_test.py | 261 ++
.../transforms/write_ptransform_test.py | 126 +
sdks/python/apache_beam/typehints/__init__.py | 22 +
sdks/python/apache_beam/typehints/decorators.py | 532 +++
sdks/python/apache_beam/typehints/opcodes.py | 334 ++
.../apache_beam/typehints/trivial_inference.py | 417 ++
.../typehints/trivial_inference_test.py | 151 +
sdks/python/apache_beam/typehints/typecheck.py | 178 +
.../typehints/typed_pipeline_test.py | 251 +
sdks/python/apache_beam/typehints/typehints.py | 1062 +++++
.../apache_beam/typehints/typehints_test.py | 1062 +++++
sdks/python/apache_beam/utils/__init__.py | 22 +
sdks/python/apache_beam/utils/annotations.py | 103 +
.../apache_beam/utils/annotations_test.py | 126 +
sdks/python/apache_beam/utils/counters.pxd | 30 +
sdks/python/apache_beam/utils/counters.py | 183 +
sdks/python/apache_beam/utils/path.py | 47 +
sdks/python/apache_beam/utils/path_test.py | 70 +
.../apache_beam/utils/pipeline_options.py | 557 +++
.../apache_beam/utils/pipeline_options_test.py | 192 +
.../utils/pipeline_options_validator.py | 199 +
.../utils/pipeline_options_validator_test.py | 342 ++
sdks/python/apache_beam/utils/processes.py | 52 +
sdks/python/apache_beam/utils/processes_test.py | 106 +
sdks/python/apache_beam/utils/profiler.py | 148 +
sdks/python/apache_beam/utils/proto_utils.py | 54 +
sdks/python/apache_beam/utils/retry.py | 207 +
sdks/python/apache_beam/utils/retry_test.py | 221 +
sdks/python/apache_beam/utils/timestamp.py | 213 +
sdks/python/apache_beam/utils/timestamp_test.py | 168 +
sdks/python/apache_beam/utils/urns.py | 24 +
.../python/apache_beam/utils/windowed_value.pxd | 38 +
sdks/python/apache_beam/utils/windowed_value.py | 122 +
.../apache_beam/utils/windowed_value_test.py | 71 +
sdks/python/apache_beam/version.py | 57 +
sdks/python/generate_pydoc.sh | 80 +
sdks/python/pom.xml | 191 +
sdks/python/run_postcommit.sh | 102 +
sdks/python/run_pylint.sh | 52 +
sdks/python/setup.cfg | 27 +
sdks/python/setup.py | 155 +
sdks/python/test_config.py | 44 +
sdks/python/tox.ini | 89 +
1339 files changed, 138577 insertions(+), 31086 deletions(-)
----------------------------------------------------------------------