You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ke...@apache.org on 2016/12/21 20:23:58 UTC
[50/50] incubator-beam git commit: This closes #1663: Merge master
(b3de17b) into gearpump-runner
This closes #1663: Merge master (b3de17b) into gearpump-runner
Adjustments in gearpump-runner:
[BEAM-79] Upgrade to beam-0.5.0-incubating-SNAPSHOT
[BEAM-79] Update to latest Gearpump API
>From master:
Disable automatic archiving of Maven builds
[BEAM-59] initial interfaces and classes of Beam FileSystem.
Change counter name in TestDataflowRunner
More escaping in Jenkins timestamp spec
Add RunnableOnService test for Metrics
Fix seed job fetch spec
Show timestamps on log lines in Jenkins
[BEAM-1165] Fix unexpected file creation when checking dependencies
[BEAM-1178] Make naming of logger objects consistent
[BEAM-716] Fix javadoc on with* methods [BEAM-959] Improve check preconditions in JmsIO
[BEAM-716] Use AutoValue in JmsIO
Fix grammar error (repeated for)
Empty TestPipeline need not be run
[BEAM-85, BEAM-298] Make TestPipeline a JUnit Rule checking proper usage
Change counter name in TestDataflowRunner
BigQueryIO: fix streaming write, typo in API
[BEAM-853] Force streaming execution on batch pipelines for testing. Expose the adapted source.
Use empty SideInputReader, fixes NPE in SimpleDoFnRunnerTest
Test that SimpleDoFnRunner wraps exceptions in startBundle and finishBundle
Add timer support to DoFnRunner(s)
Make TimerSpec and StateSpec fields accessible
View.asMap: minor javadoc fixes
Revert "Move InMemoryTimerInternals to runners-core"
Revert "Moves DoFnAdapters to runners-core"
Revert "Removes ArgumentProvider.windowingInternals"
Revert "Removes code for wrapping DoFn as an OldDoFn"
checkstyle: missed newline in DistributionCell
Make {Metric,Counter,Distribution}Cell public
Add PTransformOverrideFactory to the Core SDK
Move ActiveWindowSet and implementations to runners-core
Update Dataflow worker to beam-master-20161216
[BEAM-1108] Remove outdated language about experimental autoscaling
[BEAM-450] Shade modules to separate paths
[BEAM-362] Port runners to runners-core AggregatoryFactory
Move InMemoryTimerInternals to runners-core
Delete deprecated TimerCallback
Remove deprecated methods of InMemoryTimerInternals
Don't incorrectly log error in MetricsEnvironment
Renames ParDo.getNewFn to getFn
Moves DoFnAdapters to runners-core
Removes unused code from NoOpOldDoFn
Removes ArgumentProvider.windowingInternals
Removes code for wrapping DoFn as an OldDoFn
Removes OldDoFn from ParDo
Pushes uses of OldDoFn deeper inside Flink runner
Remove ParDo.of(OldDoFn) from Apex runner
Converts all easy OldDoFns to DoFn
[BEAM-1022] Add testing coverage for BigQuery streaming writes
Fix mvn command args in Apex postcommit Jenkins job
[BEAM-932] Enable findbugs validation (and fix existing issues)
Fail to split in FileBasedSource if filePattern expands to empty.
[BEAM-1154] Get side input from proper window in ReduceFn
[BEAM-1153] GcsUtil: use non-batch API for single file size requests.
Fix NPE in StatefulParDoEvaluatorFactoryTest mocking
[BEAM-1033] Retry Bigquery Verifier when Query Fails
Implement GetDefaultOutputCoder in DirectGroupByKey
SimpleDoFnRunner observes window if SideInputReader is nonempty
Better comments and cleanup
Allow empty string value for ValueProvider types.
starter: fix typo in pom.xml
Revert "Allow stateful DoFn in DataflowRunner"
Re-exclude UsesStatefulParDo tests for Dataflow
Some minor changes and fixes for sorter module
[BEAM-1149] Explode windows when fn uses side inputs
Add Jenkins postcommit for RunnableOnService in Apex runner
Update version from 0.5.0-SNAPSHOT to 0.5.0-incubating-SNAPSHOT
Update Maven Archetype versions after cutting the release branch
Move PerKeyCombineFnRunner to runners-core
Update Dataflow worker to beam-master-20161212
[maven-release-plugin] prepare for next development iteration
[maven-release-plugin] prepare branch release-0.4.0-incubating
Fix version of Kryo in examples/java jenkins-precommit profile
Revert 91cc606 "This closes #1586": Kryo + UBRFBS
[BEAM-909] improve starter archetype
Fix JDom malformed comment in Apex runner.
[BEAM-927] Fix findbugs and re-enable Maven plugin in JmsIO
[BEAM-807] Replace OldDoFn with DoFn.
[BEAM-757] Use DoFnRunner in the implementation of DoFn via FlatMapFunction.
FileBasedSinkTest: fix tests in Windows OS by using IOChannelUtils.resolve().
FileBasedSink: ignore exceptions when removing temp output files for issues in Windows OS.
[BEAM-1142] Upgrade maven-invoker to address maven bug ARCHETYPE-488.
Add Tests for Kryo Serialization of URFBS
Add no-arg constructor for UnboundedReadFromBoundedSource
Revise WindowedWordCount for runner and execution mode portability
Factor out ShardedFile from FileChecksumMatcher
Add IntervalWindow coder to the standard registry
Stop expanding PValues in DirectRunner visitors
Migrate AppliedPTransform to use AutoValue
Enable and fix DirectRunnerTest case missing @Test
[BEAM-1130] SparkRunner ResumeFromCheckpointStreamingTest Failing.
[BEAM-1133] Add maxNumRecords per micro-batch for Spark runner options.
BigQueryIO.Write: support runtime schema and table
Fix handling of null ValueProviders in DisplayData
[BEAM-551] Fix handling of default for VP
[BEAM-1120] Move some DataflowRunner configurations from code to properties
[BEAM-551] Fix toString for FileBasedSource
[BEAM-921] spark-runner: register sources and coders to serialize with java serializer
[BEAM-551] Fix handling of TextIO.Sink
...
Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4c445dd0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4c445dd0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4c445dd0
Branch: refs/heads/gearpump-runner
Commit: 4c445dd0b6de0f5045c02579cb432da4fbc5d486
Parents: b6e7bb6 647034c
Author: Kenneth Knowles <kl...@google.com>
Authored: Wed Dec 21 12:20:11 2016 -0800
Committer: Kenneth Knowles <kl...@google.com>
Committed: Wed Dec 21 12:20:11 2016 -0800
----------------------------------------------------------------------
.jenkins/common_job_properties.groovy | 7 +
...ostCommit_Java_RunnableOnService_Apex.groovy | 41 ++
README.md | 2 +-
examples/java/pom.xml | 20 +-
.../apache/beam/examples/WindowedWordCount.java | 177 +++---
.../org/apache/beam/examples/WordCount.java | 2 +-
.../examples/common/WriteWindowedFilesDoFn.java | 77 +++
.../beam/examples/complete/AutoComplete.java | 6 +-
.../apache/beam/examples/complete/TfIdf.java | 6 +-
.../examples/complete/TopWikipediaSessions.java | 15 +-
.../examples/complete/TrafficMaxLaneFlow.java | 4 +-
.../beam/examples/complete/TrafficRoutes.java | 4 +-
.../examples/cookbook/BigQueryTornadoes.java | 2 +-
.../cookbook/CombinePerKeyExamples.java | 2 +-
.../beam/examples/cookbook/FilterExamples.java | 2 +-
.../examples/cookbook/MaxPerKeyExamples.java | 2 +-
.../beam/examples/cookbook/TriggerExample.java | 4 +-
.../beam/examples/WindowedWordCountIT.java | 182 +++++-
.../examples/complete/AutoCompleteTest.java | 2 +-
examples/java8/pom.xml | 2 +-
.../beam/examples/complete/game/GameStats.java | 2 +-
.../examples/complete/game/LeaderBoard.java | 4 +-
.../beam/examples/complete/game/UserScore.java | 2 +-
.../complete/game/utils/WriteToBigQuery.java | 2 +-
.../game/utils/WriteWindowedToBigQuery.java | 7 +-
examples/pom.xml | 2 +-
pom.xml | 5 +-
runners/apex/pom.xml | 66 ++-
.../apache/beam/runners/apex/ApexRunner.java | 111 ++--
.../beam/runners/apex/ApexRunnerResult.java | 50 +-
.../beam/runners/apex/ApexYarnLauncher.java | 395 +++++++++++++
.../translation/ApexPipelineTranslator.java | 2 +
.../translation/ParDoBoundMultiTranslator.java | 22 +-
.../apex/translation/ParDoBoundTranslator.java | 22 +-
.../apex/translation/TranslationContext.java | 4 +-
.../apex/translation/WindowBoundTranslator.java | 78 +++
.../operators/ApexGroupByKeyOperator.java | 4 +-
.../operators/ApexParDoOperator.java | 27 +-
.../beam/runners/apex/ApexYarnLauncherTest.java | 138 +++++
.../FlattenPCollectionTranslatorTest.java | 15 +-
.../translation/GroupByKeyTranslatorTest.java | 21 +-
.../translation/ParDoBoundTranslatorTest.java | 38 +-
.../translation/ReadUnboundTranslatorTest.java | 15 +-
runners/core-java/pom.xml | 6 +-
.../beam/runners/core/ActiveWindowSet.java | 173 ++++++
.../apache/beam/runners/core/AssignWindows.java | 46 --
.../apache/beam/runners/core/DoFnRunner.java | 24 +-
.../apache/beam/runners/core/DoFnRunners.java | 2 -
.../core/GroupAlsoByWindowViaWindowSetDoFn.java | 12 +-
.../GroupAlsoByWindowsViaOutputBufferDoFn.java | 48 +-
.../core/GroupByKeyViaGroupByKeyOnly.java | 8 +-
.../apache/beam/runners/core/KeyedWorkItem.java | 44 ++
.../beam/runners/core/KeyedWorkItemCoder.java | 130 +++++
.../beam/runners/core/KeyedWorkItems.java | 122 ++++
.../core/LateDataDroppingDoFnRunner.java | 34 +-
.../runners/core/MergingActiveWindowSet.java | 428 ++++++++++++++
.../runners/core/NonMergingActiveWindowSet.java | 91 +++
.../beam/runners/core/PaneInfoTracker.java | 4 +
.../runners/core/PerKeyCombineFnRunner.java | 149 +++++
.../runners/core/PerKeyCombineFnRunners.java | 1 -
.../core/PushbackSideInputDoFnRunner.java | 31 +-
.../runners/core/ReduceFnContextFactory.java | 26 +-
.../beam/runners/core/ReduceFnRunner.java | 496 ++++++++++------
.../beam/runners/core/SimpleDoFnRunner.java | 252 ++++++++-
.../beam/runners/core/SimpleOldDoFnRunner.java | 9 +-
.../beam/runners/core/SplittableParDo.java | 28 +-
.../core/UnboundedReadFromBoundedSource.java | 2 +-
.../apache/beam/runners/core/WatermarkHold.java | 5 +
.../TriggerStateMachineContextFactory.java | 2 +-
.../triggers/TriggerStateMachineRunner.java | 14 +-
.../runners/core/KeyedWorkItemCoderTest.java | 64 +++
.../core/LateDataDroppingDoFnRunnerTest.java | 3 +
.../core/MergingActiveWindowSetTest.java | 262 +++++++++
.../core/PushbackSideInputDoFnRunnerTest.java | 55 +-
.../beam/runners/core/ReduceFnRunnerTest.java | 133 ++---
.../beam/runners/core/ReduceFnTester.java | 77 ++-
.../beam/runners/core/SimpleDoFnRunnerTest.java | 301 ++++++++++
.../beam/runners/core/SplittableParDoTest.java | 2 -
.../triggers/TriggerStateMachineTester.java | 25 +-
runners/direct-java/pom.xml | 3 +-
.../runners/direct/AggregatorContainer.java | 2 +-
.../CopyOnAccessInMemoryStateInternals.java | 467 +++++++++++++++
.../runners/direct/DirectExecutionContext.java | 1 -
...ectGBKIntoKeyedWorkItemsOverrideFactory.java | 9 +-
.../beam/runners/direct/DirectGraphVisitor.java | 23 +-
.../beam/runners/direct/DirectGroupByKey.java | 46 +-
.../direct/DirectGroupByKeyOverrideFactory.java | 3 +-
.../beam/runners/direct/DirectRunner.java | 7 +-
.../beam/runners/direct/EvaluationContext.java | 1 -
.../direct/ExecutorServiceParallelExecutor.java | 4 +-
.../runners/direct/ForwardingPTransform.java | 6 +-
.../GroupAlsoByWindowEvaluatorFactory.java | 8 +-
.../direct/GroupByKeyOnlyEvaluatorFactory.java | 4 +-
.../direct/KeyedPValueTrackingVisitor.java | 4 +-
.../direct/PTransformOverrideFactory.java | 35 --
.../beam/runners/direct/ParDoEvaluator.java | 1 -
.../runners/direct/ParDoEvaluatorFactory.java | 2 +-
.../direct/ParDoMultiOverrideFactory.java | 29 +-
.../ParDoSingleViaMultiOverrideFactory.java | 17 +-
...littableProcessElementsEvaluatorFactory.java | 2 +-
.../direct/StatefulParDoEvaluatorFactory.java | 4 +-
.../runners/direct/StepTransformResult.java | 1 -
.../direct/TestStreamEvaluatorFactory.java | 7 +-
.../beam/runners/direct/TransformResult.java | 1 -
.../runners/direct/ViewEvaluatorFactory.java | 8 +-
.../direct/WriteWithShardingFactory.java | 10 +-
.../direct/BoundedReadEvaluatorFactoryTest.java | 18 +-
.../runners/direct/CommittedResultTest.java | 2 +-
.../CopyOnAccessInMemoryStateInternalsTest.java | 562 +++++++++++++++++++
.../runners/direct/DirectGraphVisitorTest.java | 71 ++-
.../beam/runners/direct/DirectGraphs.java | 35 ++
.../beam/runners/direct/DirectRunnerTest.java | 36 --
.../runners/direct/EvaluationContextTest.java | 83 +--
.../direct/FlattenEvaluatorFactoryTest.java | 15 +-
.../direct/ForwardingPTransformTest.java | 4 +-
.../direct/GroupByKeyEvaluatorFactoryTest.java | 6 +-
.../GroupByKeyOnlyEvaluatorFactoryTest.java | 7 +-
.../ImmutabilityEnforcementFactoryTest.java | 2 +-
.../direct/KeyedPValueTrackingVisitorTest.java | 4 +-
.../beam/runners/direct/ParDoEvaluatorTest.java | 3 +-
.../StatefulParDoEvaluatorFactoryTest.java | 11 +-
.../runners/direct/StepTransformResultTest.java | 2 +-
.../direct/TestStreamEvaluatorFactoryTest.java | 14 +-
.../runners/direct/TransformExecutorTest.java | 9 +-
.../UnboundedReadEvaluatorFactoryTest.java | 24 +-
.../direct/ViewEvaluatorFactoryTest.java | 4 +-
.../direct/WatermarkCallbackExecutorTest.java | 6 +-
.../runners/direct/WatermarkManagerTest.java | 237 ++++----
.../direct/WriteWithShardingFactoryTest.java | 4 +-
runners/flink/examples/pom.xml | 14 +-
.../beam/runners/flink/examples/TFIDF.java | 6 +-
.../beam/runners/flink/examples/WordCount.java | 2 +-
.../flink/examples/streaming/AutoComplete.java | 6 +-
runners/flink/pom.xml | 2 +-
runners/flink/runner/pom.xml | 10 +-
.../FlinkPipelineExecutionEnvironment.java | 12 +
.../runners/flink/FlinkPipelineOptions.java | 5 +
.../apache/beam/runners/flink/FlinkRunner.java | 24 +-
.../FlinkBatchPipelineTranslator.java | 4 +-
.../FlinkBatchTransformTranslators.java | 59 +-
.../FlinkStreamingPipelineTranslator.java | 7 +-
.../FlinkStreamingTransformTranslators.java | 77 ++-
.../functions/FlinkDoFnFunction.java | 10 +-
.../FlinkMergingNonShuffleReduceFunction.java | 2 +-
.../FlinkMergingPartialReduceFunction.java | 2 +-
.../functions/FlinkMergingReduceFunction.java | 2 +-
.../functions/FlinkMultiOutputDoFnFunction.java | 10 +-
.../functions/FlinkPartialReduceFunction.java | 2 +-
.../functions/FlinkProcessContextBase.java | 21 +-
.../functions/FlinkReduceFunction.java | 2 +-
.../wrappers/streaming/DoFnOperator.java | 46 +-
.../streaming/SingletonKeyedWorkItem.java | 2 +-
.../streaming/SingletonKeyedWorkItemCoder.java | 4 +-
.../wrappers/streaming/WindowDoFnOperator.java | 14 +-
.../wrappers/streaming/WorkItemKeySelector.java | 2 +-
.../beam/runners/flink/PipelineOptionsTest.java | 6 +-
.../flink/streaming/DoFnOperatorTest.java | 13 +-
.../flink/streaming/GroupByNullKeyTest.java | 18 +-
.../streaming/TopWikipediaSessionsITCase.java | 10 +-
runners/gearpump/pom.xml | 2 +-
.../beam/runners/gearpump/GearpumpRunner.java | 2 +-
.../gearpump/examples/StreamingWordCount.java | 14 +-
.../translators/ParDoBoundMultiTranslator.java | 2 +-
.../translators/ParDoBoundTranslator.java | 2 +-
.../translators/TranslationContext.java | 3 +-
.../translators/utils/DoFnRunnerFactory.java | 2 +-
.../utils/NoOpAggregatorFactory.java | 2 +-
runners/google-cloud-dataflow-java/pom.xml | 12 +-
.../beam/runners/dataflow/DataflowClient.java | 140 +++++
.../runners/dataflow/DataflowPipelineJob.java | 31 +-
.../dataflow/DataflowPipelineTranslator.java | 15 +-
.../beam/runners/dataflow/DataflowRunner.java | 136 +++--
.../runners/dataflow/DataflowRunnerInfo.java | 92 +++
.../dataflow/internal/AssignWindows.java | 10 +-
.../DataflowUnboundedReadFromBoundedSource.java | 2 +-
.../DataflowPipelineWorkerPoolOptions.java | 57 +-
.../options/DataflowProfilingOptions.java | 8 +-
.../dataflow/testing/TestDataflowRunner.java | 35 +-
.../dataflow/util/DataflowTemplateJob.java | 2 +-
.../runners/dataflow/util/MonitoringUtil.java | 22 +-
.../beam/runners/dataflow/dataflow.properties | 23 +
.../dataflow/DataflowPipelineJobTest.java | 49 +-
.../DataflowPipelineTranslatorTest.java | 61 +-
.../dataflow/DataflowRunnerInfoTest.java | 51 ++
.../runners/dataflow/DataflowRunnerTest.java | 285 ++++++----
.../options/DataflowProfilingOptionsTest.java | 6 +-
.../testing/TestDataflowRunnerTest.java | 52 +-
.../transforms/DataflowGroupByKeyTest.java | 18 +-
.../dataflow/transforms/DataflowViewTest.java | 20 +-
.../dataflow/util/MonitoringUtilTest.java | 21 +-
runners/pom.xml | 2 +-
runners/spark/README.md | 59 +-
runners/spark/pom.xml | 55 +-
.../beam/runners/spark/EvaluationResult.java | 67 ---
.../runners/spark/SparkPipelineOptions.java | 10 +
.../beam/runners/spark/SparkPipelineResult.java | 193 +++++++
.../apache/beam/runners/spark/SparkRunner.java | 130 +++--
.../beam/runners/spark/TestSparkRunner.java | 91 ++-
.../spark/aggregators/AccumulatorSingleton.java | 6 +-
.../spark/aggregators/SparkAggregators.java | 126 +++++
.../metrics/WithNamedAggregatorsSupport.java | 4 +-
.../coders/BeamSparkRunnerRegistrator.java | 60 +-
.../beam/runners/spark/examples/WordCount.java | 15 +-
.../apache/beam/runners/spark/io/ConsoleIO.java | 2 +-
.../beam/runners/spark/io/CreateStream.java | 2 +-
.../beam/runners/spark/io/SourceDStream.java | 21 +-
.../runners/spark/io/SparkUnboundedSource.java | 17 +-
.../beam/runners/spark/io/hadoop/HadoopIO.java | 4 +-
.../runners/spark/translation/DoFnFunction.java | 110 ++--
.../spark/translation/EvaluationContext.java | 131 +----
.../translation/GroupCombineFunctions.java | 23 +-
.../spark/translation/MultiDoFnFunction.java | 135 ++---
.../spark/translation/SparkAssignWindowFn.java | 69 +++
.../spark/translation/SparkContextFactory.java | 2 +-
.../translation/SparkGroupAlsoByWindowFn.java | 214 +++++++
.../spark/translation/SparkProcessContext.java | 385 +++----------
.../spark/translation/SparkRuntimeContext.java | 62 +-
.../translation/StorageLevelPTransform.java | 2 +-
.../spark/translation/TransformTranslator.java | 67 +--
.../spark/translation/TranslationUtils.java | 33 ++
.../spark/translation/WindowingHelpers.java | 3 +-
.../streaming/StreamingTransformTranslator.java | 45 +-
.../util/SinglePrimitiveOutputPTransform.java | 2 +-
.../beam/runners/spark/ForceStreamingTest.java | 123 ++++
.../runners/spark/ProvidedSparkContextTest.java | 6 +-
.../runners/spark/SparkPipelineStateTest.java | 217 +++++++
.../spark/aggregators/ClearAggregatorsRule.java | 37 ++
.../metrics/sink/ClearAggregatorsRule.java | 33 --
.../metrics/sink/NamedAggregatorsTest.java | 3 +-
.../coders/BeamSparkRunnerRegistratorTest.java | 57 ++
.../beam/runners/spark/io/AvroPipelineTest.java | 2 +-
.../beam/runners/spark/io/NumShardsTest.java | 2 +-
.../io/hadoop/HadoopFileFormatPipelineTest.java | 2 +-
.../spark/translation/SideEffectsTest.java | 59 --
.../streaming/EmptyStreamAssertionTest.java | 4 +
.../streaming/KafkaStreamingTest.java | 34 +-
.../ResumeFromCheckpointStreamingTest.java | 32 +-
.../streaming/utils/PAssertStreaming.java | 31 +-
sdks/java/build-tools/pom.xml | 2 +-
.../src/main/resources/beam/findbugs-filter.xml | 68 ++-
sdks/java/core/pom.xml | 2 +-
.../beam/sdk/AggregatorPipelineExtractor.java | 3 +-
.../beam/sdk/annotations/Experimental.java | 5 +-
.../apache/beam/sdk/coders/CoderRegistry.java | 2 +
.../java/org/apache/beam/sdk/io/AvroIO.java | 4 +-
.../sdk/io/BoundedReadFromUnboundedSource.java | 16 +-
.../org/apache/beam/sdk/io/CountingInput.java | 52 +-
.../org/apache/beam/sdk/io/CountingSource.java | 17 +-
.../org/apache/beam/sdk/io/FileBasedSink.java | 77 ++-
.../org/apache/beam/sdk/io/FileBasedSource.java | 12 +-
.../java/org/apache/beam/sdk/io/FileSystem.java | 29 +
.../apache/beam/sdk/io/FileSystemRegistrar.java | 49 ++
.../org/apache/beam/sdk/io/FileSystems.java | 155 +++++
.../org/apache/beam/sdk/io/LocalFileSystem.java | 27 +
.../beam/sdk/io/LocalFileSystemRegistrar.java | 41 ++
.../java/org/apache/beam/sdk/io/PubsubIO.java | 4 +-
.../apache/beam/sdk/io/PubsubUnboundedSink.java | 6 +-
.../beam/sdk/io/PubsubUnboundedSource.java | 12 +-
.../main/java/org/apache/beam/sdk/io/Read.java | 4 +-
.../java/org/apache/beam/sdk/io/TextIO.java | 4 +-
.../main/java/org/apache/beam/sdk/io/Write.java | 2 +-
.../apache/beam/sdk/io/range/ByteKeyRange.java | 4 +-
.../beam/sdk/io/range/ByteKeyRangeTracker.java | 8 +-
.../apache/beam/sdk/metrics/CounterCell.java | 10 +-
.../beam/sdk/metrics/DistributionCell.java | 14 +-
.../org/apache/beam/sdk/metrics/MetricCell.java | 2 +-
.../org/apache/beam/sdk/metrics/MetricName.java | 3 +-
.../org/apache/beam/sdk/metrics/Metrics.java | 5 +-
.../beam/sdk/metrics/MetricsEnvironment.java | 8 +-
.../sdk/options/PipelineOptionsFactory.java | 110 +++-
.../beam/sdk/options/StreamingOptions.java | 2 +-
.../apache/beam/sdk/options/ValueProvider.java | 27 +-
.../sdk/runners/PTransformOverrideFactory.java | 41 ++
.../apache/beam/sdk/runners/PipelineRunner.java | 2 +-
.../beam/sdk/runners/TransformHierarchy.java | 26 +-
.../beam/sdk/testing/BigqueryMatcher.java | 48 +-
.../beam/sdk/testing/FileChecksumMatcher.java | 114 +---
.../apache/beam/sdk/testing/GatherAllPanes.java | 2 +-
.../org/apache/beam/sdk/testing/PAssert.java | 12 +-
.../apache/beam/sdk/testing/TestPipeline.java | 225 +++++++-
.../org/apache/beam/sdk/testing/TestStream.java | 2 +-
.../apache/beam/sdk/testing/UsesMetrics.java | 24 +
.../beam/sdk/testing/UsesTimersInParDo.java | 25 +
.../sdk/transforms/AggregatorRetriever.java | 2 +-
.../beam/sdk/transforms/AppliedPTransform.java | 75 +--
.../beam/sdk/transforms/ApproximateUnique.java | 4 +-
.../org/apache/beam/sdk/transforms/Combine.java | 14 +-
.../beam/sdk/transforms/CombineWithContext.java | 2 +-
.../org/apache/beam/sdk/transforms/Count.java | 2 +-
.../org/apache/beam/sdk/transforms/Create.java | 4 +-
.../apache/beam/sdk/transforms/Distinct.java | 4 +-
.../org/apache/beam/sdk/transforms/DoFn.java | 22 +
.../beam/sdk/transforms/DoFnAdapters.java | 12 +
.../apache/beam/sdk/transforms/DoFnTester.java | 29 +-
.../org/apache/beam/sdk/transforms/Filter.java | 2 +-
.../beam/sdk/transforms/FlatMapElements.java | 2 +-
.../org/apache/beam/sdk/transforms/Flatten.java | 4 +-
.../apache/beam/sdk/transforms/GroupByKey.java | 2 +-
.../org/apache/beam/sdk/transforms/Keys.java | 2 +-
.../org/apache/beam/sdk/transforms/KvSwap.java | 2 +-
.../org/apache/beam/sdk/transforms/Latest.java | 4 +-
.../apache/beam/sdk/transforms/MapElements.java | 2 +-
.../apache/beam/sdk/transforms/PTransform.java | 4 +-
.../org/apache/beam/sdk/transforms/ParDo.java | 191 +------
.../apache/beam/sdk/transforms/Partition.java | 2 +-
.../org/apache/beam/sdk/transforms/Regex.java | 14 +-
.../org/apache/beam/sdk/transforms/Sample.java | 2 +-
.../org/apache/beam/sdk/transforms/Values.java | 2 +-
.../org/apache/beam/sdk/transforms/View.java | 16 +-
.../apache/beam/sdk/transforms/WithKeys.java | 2 +-
.../beam/sdk/transforms/WithTimestamps.java | 2 +-
.../sdk/transforms/display/DisplayData.java | 22 +
.../beam/sdk/transforms/join/CoGroupByKey.java | 2 +-
.../reflect/ByteBuddyDoFnInvokerFactory.java | 11 +
.../sdk/transforms/reflect/DoFnInvoker.java | 8 +
.../sdk/transforms/reflect/DoFnSignature.java | 26 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 96 +++-
.../beam/sdk/transforms/windowing/Window.java | 4 +-
.../apache/beam/sdk/util/ActiveWindowSet.java | 173 ------
.../org/apache/beam/sdk/util/ApiSurface.java | 52 +-
.../apache/beam/sdk/util/ExecutionContext.java | 8 +-
.../beam/sdk/util/ExplicitShardedFile.java | 120 ++++
.../java/org/apache/beam/sdk/util/GcsUtil.java | 107 +++-
.../org/apache/beam/sdk/util/KeyedWorkItem.java | 43 --
.../beam/sdk/util/KeyedWorkItemCoder.java | 128 -----
.../apache/beam/sdk/util/KeyedWorkItems.java | 121 ----
.../beam/sdk/util/MergingActiveWindowSet.java | 428 --------------
.../sdk/util/NonMergingActiveWindowSet.java | 91 ---
.../beam/sdk/util/NumberedShardedFile.java | 220 ++++++++
.../beam/sdk/util/PerKeyCombineFnRunner.java | 148 -----
.../org/apache/beam/sdk/util/PropertyNames.java | 2 +
.../org/apache/beam/sdk/util/ReleaseInfo.java | 4 -
.../org/apache/beam/sdk/util/Reshuffle.java | 2 +-
.../org/apache/beam/sdk/util/ShardedFile.java | 42 ++
.../CopyOnAccessInMemoryStateInternals.java | 453 ---------------
.../sdk/util/state/InMemoryStateInternals.java | 33 +-
.../sdk/util/state/InMemoryTimerInternals.java | 126 +++--
.../beam/sdk/util/state/StateInternals.java | 2 +-
.../beam/sdk/util/state/TimerCallback.java | 35 --
.../org/apache/beam/sdk/values/PCollection.java | 4 +-
.../beam/sdk/values/PCollectionTuple.java | 2 +-
.../org/apache/beam/sdk/values/TypedPValue.java | 2 +-
.../sdk/AggregatorPipelineExtractorTest.java | 8 +-
.../java/org/apache/beam/sdk/PipelineTest.java | 6 +-
.../beam/sdk/coders/CoderRegistryTest.java | 4 +-
.../apache/beam/sdk/io/CountingInputTest.java | 63 ++-
.../apache/beam/sdk/io/CountingSourceTest.java | 10 +
.../apache/beam/sdk/io/FileBasedSinkTest.java | 11 +-
.../apache/beam/sdk/io/FileBasedSourceTest.java | 26 +
.../org/apache/beam/sdk/io/FileSystemsTest.java | 104 ++++
.../sdk/io/LocalFileSystemRegistrarTest.java | 44 ++
.../java/org/apache/beam/sdk/io/TextIOTest.java | 19 +
.../java/org/apache/beam/sdk/io/WriteTest.java | 2 +-
.../org/apache/beam/sdk/io/XmlSinkTest.java | 4 +-
.../apache/beam/sdk/metrics/MetricMatchers.java | 47 +-
.../apache/beam/sdk/metrics/MetricsTest.java | 63 ++-
.../sdk/options/PipelineOptionsFactoryTest.java | 222 +++++++-
.../beam/sdk/options/ValueProviderTest.java | 21 +
.../sdk/runners/TransformHierarchyTest.java | 17 +-
.../beam/sdk/runners/TransformTreeTest.java | 4 +-
.../beam/sdk/testing/BigqueryMatcherTest.java | 82 +--
.../sdk/testing/FileChecksumMatcherTest.java | 77 ---
.../beam/sdk/testing/TestPipelineTest.java | 186 +++++-
.../beam/sdk/transforms/DoFnTesterTest.java | 2 +-
.../sdk/transforms/FlatMapElementsTest.java | 2 +-
.../beam/sdk/transforms/GroupByKeyTest.java | 4 +-
.../beam/sdk/transforms/MapElementsTest.java | 2 +-
.../apache/beam/sdk/transforms/NoOpOldDoFn.java | 72 ---
.../apache/beam/sdk/transforms/OldDoFnTest.java | 125 ++---
.../beam/sdk/transforms/PTransformTest.java | 2 +-
.../beam/sdk/transforms/ParDoLifecycleTest.java | 28 +-
.../apache/beam/sdk/transforms/ParDoTest.java | 342 ++++++-----
.../beam/sdk/transforms/SplittableDoFnTest.java | 58 +-
.../apache/beam/sdk/transforms/ViewTest.java | 4 +-
.../display/DisplayDataEvaluatorTest.java | 2 +-
.../display/DisplayDataMatchersTest.java | 6 +-
.../sdk/transforms/display/DisplayDataTest.java | 90 ++-
.../DoFnSignaturesSplittableDoFnTest.java | 3 +-
.../transforms/reflect/DoFnSignaturesTest.java | 47 ++
.../sdk/transforms/windowing/WindowingTest.java | 2 +-
.../org/apache/beam/sdk/util/GcsUtilTest.java | 85 ++-
.../beam/sdk/util/KeyedWorkItemCoderTest.java | 62 --
.../sdk/util/MergingActiveWindowSetTest.java | 262 ---------
.../beam/sdk/util/NumberedShardedFileTest.java | 181 ++++++
.../apache/beam/sdk/util/StringUtilsTest.java | 6 +-
.../CopyOnAccessInMemoryStateInternalsTest.java | 552 ------------------
.../util/state/InMemoryTimerInternalsTest.java | 125 +++--
.../org/apache/beam/sdk/values/PDoneTest.java | 4 +-
sdks/java/extensions/join-library/pom.xml | 2 +-
sdks/java/extensions/pom.xml | 2 +-
sdks/java/extensions/sorter/README.md | 2 +-
sdks/java/extensions/sorter/pom.xml | 10 +-
.../sorter/BufferedExternalSorter.java | 6 +-
.../sdk/extensions/sorter/ExternalSorter.java | 15 +-
.../sdk/extensions/sorter/InMemorySorter.java | 26 +-
.../beam/sdk/extensions/sorter/SortValues.java | 2 +-
.../sorter/BufferedExternalSorterTest.java | 16 +
.../extensions/sorter/ExternalSorterTest.java | 16 +
.../extensions/sorter/InMemorySorterTest.java | 8 +
sdks/java/io/google-cloud-platform/pom.xml | 8 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 453 +++++++++++----
.../sdk/io/gcp/bigquery/BigQueryServices.java | 7 +-
.../io/gcp/bigquery/BigQueryServicesImpl.java | 121 +++-
.../io/gcp/bigquery/BigQueryTableInserter.java | 217 -------
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 28 +-
.../io/gcp/bigtable/BigtableServiceImpl.java | 6 +-
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 4 +-
.../beam/sdk/io/gcp/storage/GcsFileSystem.java | 34 ++
.../io/gcp/storage/GcsFileSystemRegistrar.java | 42 ++
.../beam/sdk/io/gcp/storage/package-info.java | 21 +
.../apache/beam/sdk/io/gcp/ApiSurfaceTest.java | 2 +
.../sdk/io/gcp/bigquery/BigQueryIOTest.java | 543 ++++++++++++++----
.../gcp/bigquery/BigQueryServicesImplTest.java | 139 ++++-
.../gcp/bigquery/BigQueryTableInserterTest.java | 245 --------
.../sdk/io/gcp/bigquery/BigQueryUtilTest.java | 50 +-
.../gcp/storage/GcsFileSystemRegistrarTest.java | 51 ++
sdks/java/io/hdfs/pom.xml | 8 +-
.../beam/sdk/io/hdfs/HadoopFileSystem.java | 29 +
.../sdk/io/hdfs/HadoopFileSystemRegistrar.java | 42 ++
.../io/hdfs/HadoopFileSystemRegistrarTest.java | 52 ++
sdks/java/io/jdbc/pom.xml | 2 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIO.java | 4 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 4 +-
sdks/java/io/jms/pom.xml | 22 +-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 342 +++++++----
.../org/apache/beam/sdk/io/jms/JmsRecord.java | 8 +-
sdks/java/io/kafka/pom.xml | 2 +-
.../org/apache/beam/sdk/io/kafka/KafkaIO.java | 20 +-
sdks/java/io/kinesis/pom.xml | 2 +-
sdks/java/io/mongodb/pom.xml | 2 +-
.../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 4 +-
.../apache/beam/sdk/io/mongodb/MongoDbIO.java | 16 +-
.../sdk/io/mongodb/MongoDBGridFSIOTest.java | 8 +-
.../beam/sdk/io/mongodb/MongoDbIOTest.java | 8 +-
sdks/java/io/pom.xml | 2 +-
sdks/java/java8tests/pom.xml | 2 +-
.../maven-archetypes/examples-java8/pom.xml | 13 +-
.../main/resources/archetype-resources/pom.xml | 122 +++-
sdks/java/maven-archetypes/examples/pom.xml | 13 +-
.../main/resources/archetype-resources/pom.xml | 47 +-
sdks/java/maven-archetypes/pom.xml | 2 +-
sdks/java/maven-archetypes/starter/pom.xml | 9 +-
.../main/resources/archetype-resources/pom.xml | 16 +-
.../resources/projects/basic/reference/pom.xml | 16 +-
sdks/java/pom.xml | 2 +-
sdks/pom.xml | 2 +-
446 files changed, 13550 insertions(+), 7606 deletions(-)
----------------------------------------------------------------------