You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ke...@apache.org on 2017/11/07 16:55:06 UTC
[50/50] [abbrv] beam git commit: This closes #4091: Merge branch
'master' upto commit 269bf89 into mr-runner
This closes #4091: Merge branch 'master' upto commit 269bf89 into mr-runner
mr-runner: re-enable sdks/python module.
mr-runner: update to 2.3.0-SNAPSHOT.
Check that bigtableWriter is non-null before calling close().
Added a preprocessing step to the Cloud Spanner sink.
Fix Repackaging Configuration in the the DirectRunner
Migrate shared Fn Execution code to Java7
update dataflow.version
[BEAM-3114] Generate text proto config properly in container boot code
[BEAM-3113] Disable stack trace optimization in java container
Fix Go package comment for syscallx
Add all portability protos to Go
[BEAM-2728] Extension for sketch-based statistics : HyperLogLog
Add a runners/java-fn-execution module
Add sdks/java/fn-execution
[BEAM-3135] Adding futures dependency to python SDK
Updates BigQueryTableSource to consider data in streaming buffer when determining estimated size.
Getting AutoValue 1.5.1 working in Beam.
Add License Header to SqlTypeUtils
[BEAM-2203] Implement TIMESTAMPADD
Fix working dir in website precommits
Do not relocate generated Model Classes
Updates Python datastore wordcount example to take a dataset parameter.
Remove obsolete extra parameter
[BEAM-2482] - CodedValueMutationDetector should use the coders structural value
Changed the mutation detector to be based on structural value only
Allows to set a Cloud Spanner host. https://batch-spanner.googleapis.com/ is set as a default host name.
[BEAM-2468] Reading Kinesis records in the background
[BEAM-3054] Uses locale-insensitive number formatting in ESIO and tests
Added VoidSerializer for KafkaIO. Modified KafkaIO.Write.values() to auto add the VoidSerializer for the key.serializer config for kafka producer
[BEAM-1542] SpannerIO: mutation encoding and size estimation improvements
[BEAM-2979] Fix a race condition in getWatermark() in KafkaIO.
[BEAM-3112] Improve error messages in ElasticsearchIO test utils
[BEAM-3111] Upgrade Elasticsearch to 5.6.3 and clean pom
Temporarily disable Dataflow pipeline_url metadata
NonNull by default in sdk/transforms/splittabledofn
NonNull by default in sdk/transforms/join
NonNull by default in sdk/transforms/display
NonNull by default in sdk/transforms/reflect
NonNull by default in sdk/transforms/windowing
NonNull by default in sdk/transforms
Reading spanner schema transform
Suppress AutoValue warnings in TextIO
Remove extraneous type arguments in Latest.java
Remove extraneous type arguments in PAssert
Remove coveralls invocations from all jobs
Adds ParseResult.failure()
Many improvements to TikaIO
Add missing @RunWith to test.
[BEAM-2566] Decouple SDK harness from Dataflow runner by elevating experiments and SDK harness configuration to java-sdk-core.
Add python_requires to setup.py
CR: [BEAM-3005] Set JVM max heap size in java container
Update rat exclusion for python and Go protos
Declare .go and Dockerfile as text in gitattributes
[BEAM-3005] Set JVM max heap size in java container
[BEAM-3005] Add resource limits to provision proto
Stage the pipeline in Python DataflowRunner
Add zip to the list of accepted extra package file types.
Fix remaining nullability errors
Increase Java postcommit timeout to 240
Manage RAT plugin more centrally; only toggle skipping
Fix RAT exclusions
Rearrange .gitignore slightly
Fix typo in seed job
Rename seed job so it is first in glob used by prior seed job
Fix typo in seed jobs
Increase seed job(s) timeout to 60 minutes
Increase seed job(s) timeout to 30 minutes
Make the main seed job standalone
Increase job_beam_PreCommit_Java_MavenInstall timeout from 2.5 to 4 hours.
Implement FnApi side inputs in Python.
Unit test for label pipeline option
NonNull by default in metrics
Ignore findbugs in AutoValue generated classes
NonNull by default for sdk/testing
NonNull by default in sdk/state
NonNull by default in sdk/runners
NonNull by default in sdk/annotations
NonNull by default in sdk/coders
Make Java core SDK root dir NonNull by default
Add dep on Apache-licensed findbugs-annotations implementation
[BEAM-2682] Deletes AvroIOTransformTest
Clone source to a distinguished subdirectory of Jenkins workspace
Adding lull tracking for python sampler
Avoids generating proto files for Windows if grpcio-tools is not installed.
Do not crash when RawPTransform has null spec
Unit test to repro NPE in PTransformTranslation
Created Java snippets file
Add standalone version of seed job
Pin runner harness also for official BEAM releases.
Remove duplicate mocking in DataflowRunnerTest
Add assertion that valid jobs must have staged pipeline
Stage the pipeline without using a temp file
Stage the portable pipeline in Dataflow
Add ability to stage explicit file list
Improve GcsFileSystem errors messages slightly
[BEAM-2720] Update kafka client version to 0.11.0.1
Update PipelineTest.testReplacedNames
DirectRunner: Replace use of RawPTransform with NotSerializable.forUrn translators
Wordcount on fnapi pipeline and IT test.
Clearer getOrDefault style in RehydratedComponents
...
Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b6f22aa7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b6f22aa7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b6f22aa7
Branch: refs/heads/mr-runner
Commit: b6f22aa76ad217fff6078e9351676895ac793ecc
Parents: 5fa0b14 202a2cb
Author: Kenneth Knowles <ke...@apache.org>
Authored: Tue Nov 7 08:53:31 2017 -0800
Committer: Kenneth Knowles <ke...@apache.org>
Committed: Tue Nov 7 08:53:31 2017 -0800
----------------------------------------------------------------------
.gitattributes | 2 +
.gitignore | 9 +-
.test-infra/jenkins/PreCommit_Pipeline.groovy | 129 +
.../jenkins/common_job_properties.groovy | 205 +-
.test-infra/jenkins/job_00_seed.groovy | 114 +
.test-infra/jenkins/job_beam_Java_Build.groovy | 82 +
.../jenkins/job_beam_Java_CodeHealth.groovy | 39 +
.../job_beam_Java_IntegrationTest.groovy | 63 +
.../jenkins/job_beam_Java_UnitTest.groovy | 48 +
...job_beam_PostCommit_Java_MavenInstall.groovy | 21 +-
..._PostCommit_Java_MavenInstall_Windows.groovy | 2 +-
...ommit_Python_ValidatesRunner_Dataflow.groovy | 54 +
.../job_beam_PreCommit_Go_MavenInstall.groovy | 56 +
.../job_beam_PreCommit_Java_MavenInstall.groovy | 21 +-
.../jenkins/job_beam_PreCommit_Pipeline.groovy | 84 +
...ob_beam_PreCommit_Python_MavenInstall.groovy | 56 +
.../job_beam_PreCommit_Website_Merge.groovy | 3 +
.../job_beam_PreCommit_Website_Stage.groovy | 3 +
.../job_beam_PreCommit_Website_Test.groovy | 3 +
.../jenkins/job_beam_Python_UnitTest.groovy | 40 +
.../job_beam_Release_NightlySnapshot.groovy | 22 +-
.test-infra/jenkins/job_seed.groovy | 53 -
.test-infra/jenkins/job_seed_standalone.groovy | 114 +
examples/java/pom.xml | 8 +-
.../apache/beam/examples/WindowedWordCount.java | 1 -
.../examples/cookbook/BigQueryTornadoes.java | 2 +-
.../cookbook/CombinePerKeyExamples.java | 2 +-
.../beam/examples/cookbook/FilterExamples.java | 2 +-
.../beam/examples/cookbook/JoinExamples.java | 6 +-
.../examples/cookbook/MaxPerKeyExamples.java | 2 +-
examples/java8/pom.xml | 14 +-
.../complete/game/injector/Injector.java | 2 +-
.../complete/game/injector/InjectorUtils.java | 2 +-
.../examples/website_snippets/Snippets.java | 87 +
.../examples/website_snippets/SnippetsTest.java | 114 +
examples/pom.xml | 2 +-
model/fn-execution/pom.xml | 114 +
.../src/main/proto/beam_fn_api.proto | 729 ++++
.../src/main/proto/beam_provision_api.proto | 99 +
.../model/fnexecution/v1/standard_coders.yaml | 195 +
model/job-management/pom.xml | 114 +
.../src/main/proto/beam_artifact_api.proto | 134 +
.../src/main/proto/beam_job_api.proto | 174 +
model/pipeline/pom.xml | 89 +
.../src/main/proto/beam_runner_api.proto | 843 +++++
model/pipeline/src/main/proto/endpoints.proto | 47 +
.../src/main/proto/standard_window_fns.proto | 54 +
model/pom.xml | 40 +
pom.xml | 368 +-
runners/apex/pom.xml | 9 +-
.../beam/runners/apex/ApexPipelineOptions.java | 2 +-
.../beam/runners/apex/ApexRunnerResult.java | 4 +-
.../apex/translation/ParDoTranslator.java | 24 +-
.../operators/ApexParDoOperator.java | 22 +-
.../translation/utils/ApexStateInternals.java | 17 +-
.../apex/translation/utils/ValuesSource.java | 4 -
.../beam/runners/apex/ApexYarnLauncherTest.java | 9 +-
.../apex/examples/UnboundedTextSource.java | 4 -
.../FlattenPCollectionTranslatorTest.java | 1 -
.../translation/GroupByKeyTranslatorTest.java | 4 -
.../apex/translation/ParDoTranslatorTest.java | 4 +-
.../translation/utils/CollectionSource.java | 4 -
runners/core-construction-java/pom.xml | 19 +-
.../construction/ArtifactServiceStager.java | 244 ++
.../core/construction/CoderTranslation.java | 22 +-
.../core/construction/CombineTranslation.java | 224 +-
.../CreatePCollectionViewTranslation.java | 28 +-
.../construction/DisplayDataTranslation.java | 4 +-
.../core/construction/FlattenTranslator.java | 12 +-
.../construction/GroupByKeyTranslation.java | 17 +-
.../construction/PCollectionTranslation.java | 10 +-
.../construction/PTransformTranslation.java | 292 +-
.../core/construction/ParDoTranslation.java | 392 +-
.../PipelineOptionsTranslation.java | 51 +
.../core/construction/PipelineTranslation.java | 137 +-
.../core/construction/ReadTranslation.java | 113 +-
.../core/construction/RehydratedComponents.java | 9 +-
.../construction/RunnerPCollectionView.java | 2 +-
.../core/construction/SdkComponents.java | 17 +-
.../core/construction/SplittableParDo.java | 8 +
.../construction/TestStreamTranslation.java | 170 +-
.../TransformPayloadTranslatorRegistrar.java | 2 +
.../core/construction/TriggerTranslation.java | 10 +-
.../construction/WindowIntoTranslation.java | 30 +-
.../WindowingStrategyTranslation.java | 150 +-
.../construction/WriteFilesTranslation.java | 245 +-
.../construction/ArtifactServiceStagerTest.java | 141 +
.../core/construction/CoderTranslationTest.java | 4 +-
.../construction/CombineTranslationTest.java | 20 +-
.../CreatePCollectionViewTranslationTest.java | 10 +-
.../InMemoryArtifactStagerService.java | 152 +
.../PCollectionTranslationTest.java | 2 +-
.../construction/PTransformMatchersTest.java | 15 +-
.../construction/PTransformTranslationTest.java | 39 +-
.../core/construction/ParDoTranslationTest.java | 9 +-
.../PipelineOptionsTranslationTest.java | 143 +
.../construction/PipelineTranslationTest.java | 2 +-
.../core/construction/ReadTranslationTest.java | 14 +-
.../core/construction/SdkComponentsTest.java | 2 +-
.../construction/TestStreamTranslationTest.java | 12 +-
.../construction/WindowIntoTranslationTest.java | 2 +-
.../WindowingStrategyTranslationTest.java | 2 +-
.../construction/WriteFilesTranslationTest.java | 15 +-
runners/core-java/pom.xml | 31 +-
.../runners/core/InMemoryStateInternals.java | 39 +-
.../runners/core/InMemoryTimerInternals.java | 28 +-
.../beam/runners/core/SideInputHandler.java | 2 +-
.../core/SplittableParDoViaKeyedWorkItems.java | 7 +
.../apache/beam/runners/core/StateTable.java | 40 +-
.../org/apache/beam/runners/core/StateTags.java | 13 +
.../core/TestInMemoryStateInternals.java | 6 +-
.../apache/beam/runners/core/WatermarkHold.java | 6 +-
.../runners/core/fn/FnApiControlClient.java | 148 +
.../core/fn/FnApiControlClientPoolService.java | 66 +
.../beam/runners/core/fn/FnDataReceiver.java | 33 +
.../beam/runners/core/fn/FnDataService.java | 81 +
.../beam/runners/core/fn/SdkHarnessClient.java | 172 +
.../runners/core/fn/SdkHarnessDoFnRunner.java | 102 +
.../beam/runners/core/fn/package-info.java | 22 +
.../core/metrics/MetricsContainerImpl.java | 40 +
.../triggers/AfterWatermarkStateMachine.java | 8 +-
.../triggers/TriggerStateMachineRunner.java | 3 +-
.../core/triggers/TriggerStateMachines.java | 2 +-
.../beam/runners/core/ReduceFnTester.java | 22 +-
.../beam/runners/core/StateInternalsTest.java | 59 +
.../fn/FnApiControlClientPoolServiceTest.java | 65 +
.../runners/core/fn/FnApiControlClientTest.java | 139 +
.../runners/core/fn/SdkHarnessClientTest.java | 96 +
.../core/fn/SdkHarnessDoFnRunnerTest.java | 73 +
.../core/metrics/MetricsContainerImplTest.java | 10 +
.../AfterWatermarkStateMachineTest.java | 29 +
.../triggers/TriggerStateMachineTester.java | 13 +-
.../core/triggers/TriggerStateMachinesTest.java | 2 +-
runners/direct-java/pom.xml | 14 +-
.../CopyOnAccessInMemoryStateInternals.java | 10 +-
.../beam/runners/direct/DirectGroupByKey.java | 16 +-
.../beam/runners/direct/DirectRunner.java | 5 +-
.../GroupAlsoByWindowEvaluatorFactory.java | 2 +-
.../beam/runners/direct/MultiStepCombine.java | 18 +-
.../runners/direct/ParDoEvaluatorFactory.java | 30 +-
.../direct/ParDoMultiOverrideFactory.java | 9 +-
...littableProcessElementsEvaluatorFactory.java | 3 +-
.../direct/StatefulParDoEvaluatorFactory.java | 15 +-
.../direct/TestStreamEvaluatorFactory.java | 9 +-
.../direct/TransformEvaluatorRegistry.java | 37 +-
.../runners/direct/ViewOverrideFactory.java | 8 +-
.../direct/WriteWithShardingFactory.java | 37 +-
.../direct/BoundedReadEvaluatorFactoryTest.java | 4 -
.../CopyOnAccessInMemoryStateInternalsTest.java | 74 +-
.../UnboundedReadEvaluatorFactoryTest.java | 3 -
.../direct/WriteWithShardingFactoryTest.java | 11 +-
runners/flink/pom.xml | 4 +-
.../runners/flink/CreateStreamingFlinkView.java | 3 +
.../flink/FlinkBatchPipelineTranslator.java | 3 +-
.../runners/flink/FlinkPipelineOptions.java | 11 +
.../FlinkStreamingTransformTranslators.java | 244 +-
.../functions/FlinkAssignContext.java | 17 +-
.../translation/types/CoderTypeSerializer.java | 2 +-
.../wrappers/streaming/DoFnOperator.java | 429 ++-
.../streaming/SplittableDoFnOperator.java | 4 +-
.../wrappers/streaming/WindowDoFnOperator.java | 4 +-
.../state/FlinkSplitStateInternals.java | 8 +-
.../streaming/state/FlinkStateInternals.java | 13 +-
.../beam/runners/flink/PipelineOptionsTest.java | 21 +-
.../flink/streaming/DoFnOperatorTest.java | 161 +-
.../flink/streaming/TestCountingSource.java | 3 -
.../types/CoderTypeSerializerTest.java | 79 +
runners/gcp/gcemd/Dockerfile | 30 +
runners/gcp/gcemd/main.go | 85 +
runners/gcp/gcemd/pom.xml | 154 +
runners/gcp/gcsproxy/Dockerfile | 30 +
runners/gcp/gcsproxy/main.go | 91 +
runners/gcp/gcsproxy/pom.xml | 154 +
runners/gcp/pom.xml | 38 +
runners/gearpump/pom.xml | 21 +-
.../gearpump/GearpumpPipelineResult.java | 3 +-
.../translators/GroupByKeyTranslator.java | 2 +-
.../gearpump/translators/io/GearpumpSource.java | 7 +-
.../gearpump/translators/io/ValuesSource.java | 8 -
.../translators/utils/DoFnRunnerFactory.java | 7 +-
.../translators/utils/TranslatorUtils.java | 20 -
.../FlattenPCollectionsTranslatorTest.java | 6 +
runners/google-cloud-dataflow-java/pom.xml | 22 +-
.../beam/runners/dataflow/AssignWindows.java | 84 -
.../dataflow/BatchStatefulParDoOverrides.java | 71 +-
.../dataflow/DataflowPipelineTranslator.java | 5 +-
.../beam/runners/dataflow/DataflowRunner.java | 69 +-
.../runners/dataflow/DataflowRunnerInfo.java | 91 +-
.../dataflow/PrimitiveParDoSingleFactory.java | 2 +-
.../dataflow/options/CloudDebuggerOptions.java | 3 +-
.../options/DataflowPipelineDebugOptions.java | 23 +-
.../options/DataflowPipelineOptions.java | 8 +
.../options/DataflowProfilingOptions.java | 3 +-
.../options/DataflowWorkerLoggingOptions.java | 6 +
...aultCoderCloudObjectTranslatorRegistrar.java | 2 +
.../beam/runners/dataflow/util/DoFnInfo.java | 104 -
.../beam/runners/dataflow/util/GcsStager.java | 53 +-
.../runners/dataflow/util/MonitoringUtil.java | 2 +-
.../beam/runners/dataflow/util/PackageUtil.java | 581 +--
.../beam/runners/dataflow/util/Stager.java | 32 +-
.../BatchStatefulParDoOverridesTest.java | 45 +-
.../DataflowPipelineTranslatorTest.java | 4 +-
.../dataflow/DataflowRunnerInfoTest.java | 9 +
.../runners/dataflow/DataflowRunnerTest.java | 107 +-
.../dataflow/util/MonitoringUtilTest.java | 4 +-
.../runners/dataflow/util/PackageUtilTest.java | 124 +-
runners/java-fn-execution/pom.xml | 91 +
.../beam/runners/fnexecution/ServerFactory.java | 104 +
.../beam/runners/fnexecution/package-info.java | 23 +
.../runners/fnexecution/ServerFactoryTest.java | 153 +
runners/local-artifact-service-java/pom.xml | 116 +
.../LocalFileSystemArtifactStagerService.java | 279 ++
.../beam/artifact/local/package-info.java | 22 +
...ocalFileSystemArtifactStagerServiceTest.java | 301 ++
runners/map-reduce/pom.xml | 2 +-
runners/pom.xml | 6 +-
runners/reference/job-server/pom.xml | 82 +
.../reference/job/ReferenceRunnerJobServer.java | 77 +
.../job/ReferenceRunnerJobService.java | 79 +
.../runners/reference/job/package-info.java | 23 +
.../job/ReferenceRunnerJobServiceTest.java | 34 +
runners/reference/pom.xml | 39 +
runners/spark/pom.xml | 6 +-
.../apache/beam/runners/spark/SparkRunner.java | 5 +-
.../beam/runners/spark/io/CreateStream.java | 104 +-
.../SparkGroupAlsoByWindowViaWindowSet.java | 819 ++--
.../spark/stateful/SparkTimerInternals.java | 21 +-
.../spark/translation/BoundedDataset.java | 17 +-
.../beam/runners/spark/translation/Dataset.java | 3 +-
.../spark/translation/EvaluationContext.java | 23 +-
.../spark/translation/SparkContextFactory.java | 2 -
.../translation/StorageLevelPTransform.java | 37 -
.../spark/translation/TransformTranslator.java | 53 +-
.../spark/translation/TranslationUtils.java | 78 +
.../streaming/StreamingTransformTranslator.java | 86 +-
.../translation/streaming/UnboundedDataset.java | 27 +-
.../streaming/WatermarkSyncedDStream.java | 149 +
.../spark/util/GlobalWatermarkHolder.java | 302 +-
.../runners/spark/SparkPipelineStateTest.java | 4 +-
.../runners/spark/SparkRunnerDebuggerTest.java | 11 +-
.../spark/translation/StorageLevelTest.java | 75 -
.../translation/streaming/CreateStreamTest.java | 33 +-
.../spark/src/test/resources/log4j.properties | 11 +-
sdks/CONTAINERS.md | 162 +
sdks/common/fn-api/pom.xml | 109 -
.../fn-api/src/main/proto/beam_fn_api.proto | 659 ----
.../org/apache/beam/fn/v1/standard_coders.yaml | 195 -
sdks/common/pom.xml | 39 -
sdks/common/runner-api/pom.xml | 109 -
.../src/main/proto/beam_job_api.proto | 143 -
.../src/main/proto/beam_runner_api.proto | 812 ----
.../src/main/proto/standard_window_fns.proto | 53 -
sdks/go/BUILD.md | 63 +
sdks/go/cmd/beamctl/cmd/artifact.go | 98 +
sdks/go/cmd/beamctl/cmd/root.go | 56 +
sdks/go/cmd/beamctl/main.go | 31 +
sdks/go/descriptor.xml | 29 +
sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go | 155 +
sdks/go/pkg/beam/artifact/gcsproxy/staging.go | 200 +
sdks/go/pkg/beam/artifact/materialize.go | 240 ++
sdks/go/pkg/beam/artifact/materialize_test.go | 238 ++
sdks/go/pkg/beam/artifact/server_test.go | 213 ++
sdks/go/pkg/beam/artifact/stage.go | 238 ++
sdks/go/pkg/beam/artifact/stage_test.go | 98 +
.../beam/model/fnexecution_v1/beam_fn_api.pb.go | 2729 ++++++++++++++
.../fnexecution_v1/beam_provision_api.pb.go | 306 ++
sdks/go/pkg/beam/model/gen.go | 22 +
.../jobmanagement_v1/beam_artifact_api.pb.go | 690 ++++
.../model/jobmanagement_v1/beam_job_api.pb.go | 903 +++++
.../model/pipeline_v1/beam_runner_api.pb.go | 3491 ++++++++++++++++++
.../pkg/beam/model/pipeline_v1/endpoints.pb.go | 160 +
.../model/pipeline_v1/standard_window_fns.pb.go | 120 +
sdks/go/pkg/beam/provision/provision_test.go | 54 +
sdks/go/pkg/beam/provision/provison.go | 80 +
sdks/go/pkg/beam/util/errorx/guarded.go | 47 +
sdks/go/pkg/beam/util/execx/exec.go | 33 +
sdks/go/pkg/beam/util/gcsx/gcs.go | 88 +
sdks/go/pkg/beam/util/grpcx/dial.go | 37 +
sdks/go/pkg/beam/util/grpcx/metadata.go | 55 +
sdks/go/pkg/beam/util/syscallx/syscall.go | 27 +
.../pkg/beam/util/syscallx/syscall_default.go | 28 +
sdks/go/pkg/beam/util/syscallx/syscall_linux.go | 38 +
sdks/go/pom.xml | 163 +
sdks/java/build-tools/pom.xml | 2 +-
.../src/main/resources/beam/findbugs-filter.xml | 51 +-
.../resources/docker/file/openjdk7/Dockerfile | 49 +
.../docker/file/openjdk7/docker-entrypoint.sh | 24 +
.../resources/docker/file/openjdk8/Dockerfile | 49 +
.../docker/file/openjdk8/docker-entrypoint.sh | 24 +
.../resources/docker/git/openjdk8/Dockerfile | 53 +
.../docker/git/openjdk8/docker-entrypoint.sh | 22 +
.../resources/docker/release/python2/Dockerfile | 21 +
sdks/java/container/Dockerfile | 28 +
sdks/java/container/boot.go | 134 +
sdks/java/container/pom.xml | 184 +
sdks/java/core/pom.xml | 15 +-
.../main/java/org/apache/beam/sdk/Pipeline.java | 12 +-
.../org/apache/beam/sdk/PipelineResult.java | 14 +-
.../beam/sdk/annotations/Experimental.java | 8 +-
.../beam/sdk/annotations/package-info.java | 4 +
.../apache/beam/sdk/coders/BeamRecordCoder.java | 111 +
.../apache/beam/sdk/coders/CoderRegistry.java | 6 +
.../apache/beam/sdk/coders/DefaultCoder.java | 17 +-
.../beam/sdk/coders/LengthPrefixCoder.java | 3 +-
.../beam/sdk/coders/SerializableCoder.java | 5 +-
.../apache/beam/sdk/coders/StructuredCoder.java | 42 +-
.../org/apache/beam/sdk/coders/VoidCoder.java | 12 +-
.../apache/beam/sdk/coders/package-info.java | 4 +
.../java/org/apache/beam/sdk/io/AvroIO.java | 544 ++-
.../java/org/apache/beam/sdk/io/AvroSink.java | 2 +-
.../java/org/apache/beam/sdk/io/AvroSource.java | 32 +-
.../apache/beam/sdk/io/BlockBasedSource.java | 27 +-
.../apache/beam/sdk/io/CompressedSource.java | 292 +-
.../org/apache/beam/sdk/io/Compression.java | 228 ++
.../org/apache/beam/sdk/io/CountingSource.java | 3 -
.../beam/sdk/io/DefaultFilenamePolicy.java | 25 +-
.../org/apache/beam/sdk/io/FileBasedSink.java | 138 +-
.../org/apache/beam/sdk/io/FileBasedSource.java | 29 +-
.../java/org/apache/beam/sdk/io/FileIO.java | 450 +++
.../apache/beam/sdk/io/GenerateSequence.java | 3 +-
.../org/apache/beam/sdk/io/LocalFileSystem.java | 9 +-
.../main/java/org/apache/beam/sdk/io/Match.java | 156 -
.../beam/sdk/io/ReadAllViaFileBasedSource.java | 100 +-
.../apache/beam/sdk/io/ReadableFileCoder.java | 50 +
.../java/org/apache/beam/sdk/io/Source.java | 2 +-
.../java/org/apache/beam/sdk/io/TFRecordIO.java | 171 +-
.../java/org/apache/beam/sdk/io/TextIO.java | 590 +--
.../java/org/apache/beam/sdk/io/TextSource.java | 117 +-
.../java/org/apache/beam/sdk/io/WriteFiles.java | 127 +-
.../apache/beam/sdk/io/WriteFilesResult.java | 81 +
.../org/apache/beam/sdk/io/package-info.java | 2 +-
.../beam/sdk/io/range/ByteKeyRangeTracker.java | 5 +
.../beam/sdk/io/range/OffsetRangeTracker.java | 22 +-
.../org/apache/beam/sdk/metrics/MetricName.java | 7 +
.../apache/beam/sdk/metrics/package-info.java | 4 +
.../DefaultPipelineOptionsRegistrar.java | 2 +
.../beam/sdk/options/ExperimentalOptions.java | 38 +
.../beam/sdk/options/PipelineOptions.java | 36 +
.../sdk/options/PipelineOptionsFactory.java | 45 +
.../beam/sdk/options/SdkHarnessOptions.java | 173 +
.../apache/beam/sdk/options/ValueProvider.java | 28 +-
.../apache/beam/sdk/options/ValueProviders.java | 15 +-
.../java/org/apache/beam/sdk/package-info.java | 4 +
.../beam/sdk/runners/TransformHierarchy.java | 19 +-
.../apache/beam/sdk/runners/package-info.java | 4 +
.../org/apache/beam/sdk/state/BagState.java | 6 +
.../apache/beam/sdk/state/CombiningState.java | 5 +
.../apache/beam/sdk/state/GroupingState.java | 12 +-
.../org/apache/beam/sdk/state/MapState.java | 20 +-
.../apache/beam/sdk/state/ReadableState.java | 6 +
.../apache/beam/sdk/state/ReadableStates.java | 4 +-
.../org/apache/beam/sdk/state/SetState.java | 10 +-
.../org/apache/beam/sdk/state/StateSpecs.java | 4 +-
.../org/apache/beam/sdk/state/package-info.java | 4 +
.../beam/sdk/testing/FileChecksumMatcher.java | 41 +-
.../org/apache/beam/sdk/testing/PAssert.java | 74 +-
.../apache/beam/sdk/testing/PaneExtractors.java | 25 +-
.../beam/sdk/testing/SourceTestUtils.java | 11 +
.../beam/sdk/testing/SuccessOrFailure.java | 24 +-
.../apache/beam/sdk/testing/TestPipeline.java | 53 +-
.../beam/sdk/testing/WindowFnTestUtils.java | 141 +-
.../apache/beam/sdk/testing/WindowSupplier.java | 4 +-
.../apache/beam/sdk/testing/package-info.java | 5 +
.../sdk/transforms/ApproximateQuantiles.java | 10 +-
.../beam/sdk/transforms/ApproximateUnique.java | 4 +-
.../org/apache/beam/sdk/transforms/Combine.java | 10 +-
.../apache/beam/sdk/transforms/CombineFns.java | 2 +
.../apache/beam/sdk/transforms/Contextful.java | 127 +
.../org/apache/beam/sdk/transforms/Create.java | 5 +-
.../apache/beam/sdk/transforms/Distinct.java | 2 +
.../apache/beam/sdk/transforms/DoFnTester.java | 27 +-
.../beam/sdk/transforms/FlatMapElements.java | 150 +-
.../org/apache/beam/sdk/transforms/Latest.java | 2 +-
.../apache/beam/sdk/transforms/MapElements.java | 77 +-
.../org/apache/beam/sdk/transforms/Max.java | 5 +-
.../org/apache/beam/sdk/transforms/Min.java | 8 +-
.../apache/beam/sdk/transforms/PTransform.java | 7 +-
.../org/apache/beam/sdk/transforms/ParDo.java | 8 +-
.../beam/sdk/transforms/Requirements.java | 61 +
.../apache/beam/sdk/transforms/Reshuffle.java | 47 +
.../org/apache/beam/sdk/transforms/Top.java | 5 +-
.../org/apache/beam/sdk/transforms/View.java | 9 +-
.../org/apache/beam/sdk/transforms/Watch.java | 82 +-
.../apache/beam/sdk/transforms/WithKeys.java | 12 +-
.../sdk/transforms/display/DisplayData.java | 25 +-
.../sdk/transforms/display/package-info.java | 4 +
.../beam/sdk/transforms/join/CoGbkResult.java | 15 +-
.../transforms/join/KeyedPCollectionTuple.java | 10 +-
.../beam/sdk/transforms/join/RawUnionValue.java | 8 +-
.../beam/sdk/transforms/join/package-info.java | 4 +
.../beam/sdk/transforms/package-info.java | 4 +
.../reflect/ByteBuddyDoFnInvokerFactory.java | 3 +-
.../sdk/transforms/reflect/DoFnInvoker.java | 50 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 1 +
.../sdk/transforms/reflect/package-info.java | 3 +
.../splittabledofn/OffsetRangeTracker.java | 5 +-
.../transforms/splittabledofn/package-info.java | 4 +
.../windowing/IncompatibleWindowException.java | 2 +-
.../MergeOverlappingIntervalWindows.java | 4 +-
.../beam/sdk/transforms/windowing/Trigger.java | 20 +-
.../beam/sdk/transforms/windowing/Window.java | 1 +
.../sdk/transforms/windowing/package-info.java | 4 +
.../org/apache/beam/sdk/util/ApiSurface.java | 2 +
.../org/apache/beam/sdk/util/CoderUtils.java | 2 +-
.../java/org/apache/beam/sdk/util/DoFnInfo.java | 104 +
.../apache/beam/sdk/util/MutationDetectors.java | 79 +-
.../beam/sdk/util/SerializableThrowable.java | 49 +
.../apache/beam/sdk/util/SerializableUtils.java | 69 +-
.../org/apache/beam/sdk/values/BeamRecord.java | 319 ++
.../apache/beam/sdk/values/BeamRecordType.java | 96 +
.../apache/beam/sdk/values/TypeDescriptors.java | 37 +-
.../java/org/apache/beam/sdk/PipelineTest.java | 51 +-
.../apache/beam/sdk/coders/AvroCoderTest.java | 35 +-
.../apache/beam/sdk/coders/CommonCoderTest.java | 4 +-
.../apache/beam/sdk/coders/VoidCoderTest.java | 11 +-
.../java/org/apache/beam/sdk/io/AvroIOTest.java | 417 ++-
.../apache/beam/sdk/io/AvroIOTransformTest.java | 324 --
.../beam/sdk/io/CompressedSourceTest.java | 96 +-
.../apache/beam/sdk/io/FileBasedSinkTest.java | 43 +-
.../apache/beam/sdk/io/FileBasedSourceTest.java | 3 -
.../java/org/apache/beam/sdk/io/FileIOTest.java | 313 ++
.../org/apache/beam/sdk/io/FileSystemsTest.java | 15 +-
.../beam/sdk/io/OffsetBasedSourceTest.java | 3 -
.../java/org/apache/beam/sdk/io/ReadTest.java | 6 -
.../java/org/apache/beam/sdk/io/SimpleSink.java | 23 +-
.../org/apache/beam/sdk/io/TFRecordIOTest.java | 35 +-
.../org/apache/beam/sdk/io/TextIOReadTest.java | 1305 ++++---
.../org/apache/beam/sdk/io/TextIOWriteTest.java | 40 +-
.../org/apache/beam/sdk/io/WriteFilesTest.java | 11 +-
.../sdk/io/range/ByteKeyRangeTrackerTest.java | 23 +
.../apache/beam/sdk/metrics/MetricsTest.java | 28 +
.../sdk/options/PipelineOptionsFactoryTest.java | 39 +
.../beam/sdk/options/PipelineOptionsTest.java | 11 +
.../sdk/options/ProxyInvocationHandlerTest.java | 4 +-
.../beam/sdk/options/SdkHarnessOptionsTest.java | 76 +
.../beam/sdk/options/ValueProviderTest.java | 38 +-
.../runners/dataflow/TestCountingSource.java | 3 -
.../sdk/testing/InterceptingUrlClassLoader.java | 57 +
.../apache/beam/sdk/testing/PAssertTest.java | 41 +-
.../beam/sdk/testing/PaneExtractorsTest.java | 7 +-
.../beam/sdk/testing/TestPipelineTest.java | 37 +-
.../transforms/ApproximateQuantilesTest.java | 528 +--
.../apache/beam/sdk/transforms/CreateTest.java | 22 +-
.../sdk/transforms/FlatMapElementsTest.java | 35 +-
.../apache/beam/sdk/transforms/FlattenTest.java | 35 +
.../beam/sdk/transforms/MapElementsTest.java | 42 +-
.../apache/beam/sdk/transforms/ParDoTest.java | 44 +-
.../apache/beam/sdk/transforms/WatchTest.java | 46 +-
.../beam/sdk/transforms/WithKeysTest.java | 30 +-
.../transforms/reflect/DoFnInvokersTest.java | 12 +-
.../sdk/transforms/windowing/WindowTest.java | 125 +-
.../beam/sdk/util/MutationDetectorsTest.java | 56 +
.../beam/sdk/util/SerializableUtilsTest.java | 60 +
.../beam/sdk/values/TypeDescriptorsTest.java | 17 +-
.../google-cloud-platform-core/pom.xml | 2 +-
.../extensions/gcp/storage/GcsFileSystem.java | 5 +-
.../java/org/apache/beam/sdk/util/GcsUtil.java | 6 +-
.../org/apache/beam/sdk/util/GcsUtilTest.java | 45 +
sdks/java/extensions/jackson/pom.xml | 2 +-
sdks/java/extensions/join-library/pom.xml | 2 +-
sdks/java/extensions/pom.xml | 4 +-
sdks/java/extensions/protobuf/pom.xml | 2 +-
sdks/java/extensions/sketching/pom.xml | 104 +
.../sketching/ApproximateDistinct.java | 573 +++
.../sdk/extensions/sketching/package-info.java | 22 +
.../sketching/ApproximateDistinctTest.java | 209 ++
sdks/java/extensions/sorter/pom.xml | 2 +-
sdks/java/extensions/sql/NOTICE | 45 +
sdks/java/extensions/sql/pom.xml | 275 ++
.../sdk/extensions/sql/BeamRecordSqlType.java | 186 +
.../apache/beam/sdk/extensions/sql/BeamSql.java | 250 ++
.../sdk/extensions/sql/BeamSqlRecordHelper.java | 217 ++
.../beam/sdk/extensions/sql/BeamSqlUdf.java | 43 +
.../extensions/sql/example/BeamSqlExample.java | 104 +
.../extensions/sql/example/package-info.java | 23 +
.../sdk/extensions/sql/impl/BeamSqlCli.java | 65 +
.../sdk/extensions/sql/impl/BeamSqlEnv.java | 135 +
.../interpreter/BeamSqlExpressionExecutor.java | 44 +
.../sql/impl/interpreter/BeamSqlFnExecutor.java | 458 +++
.../operator/BeamSqlCaseExpression.java | 64 +
.../operator/BeamSqlCastExpression.java | 132 +
.../interpreter/operator/BeamSqlExpression.java | 79 +
.../operator/BeamSqlInputRefExpression.java | 48 +
.../interpreter/operator/BeamSqlPrimitive.java | 157 +
.../operator/BeamSqlReinterpretExpression.java | 55 +
.../operator/BeamSqlUdfExpression.java | 92 +
.../operator/BeamSqlWindowEndExpression.java | 48 +
.../operator/BeamSqlWindowExpression.java | 51 +
.../operator/BeamSqlWindowStartExpression.java | 49 +
.../sql/impl/interpreter/operator/UdafImpl.java | 87 +
.../arithmetic/BeamSqlArithmeticExpression.java | 124 +
.../arithmetic/BeamSqlDivideExpression.java | 37 +
.../arithmetic/BeamSqlMinusExpression.java | 36 +
.../arithmetic/BeamSqlModExpression.java | 36 +
.../arithmetic/BeamSqlMultiplyExpression.java | 36 +
.../arithmetic/BeamSqlPlusExpression.java | 36 +
.../operator/arithmetic/package-info.java | 22 +
.../comparison/BeamSqlCompareExpression.java | 97 +
.../comparison/BeamSqlEqualsExpression.java | 49 +
.../BeamSqlGreaterThanExpression.java | 49 +
.../BeamSqlGreaterThanOrEqualsExpression.java | 49 +
.../comparison/BeamSqlIsNotNullExpression.java | 54 +
.../comparison/BeamSqlIsNullExpression.java | 54 +
.../comparison/BeamSqlLessThanExpression.java | 49 +
.../BeamSqlLessThanOrEqualsExpression.java | 49 +
.../comparison/BeamSqlNotEqualsExpression.java | 49 +
.../operator/comparison/package-info.java | 22 +
.../date/BeamSqlCurrentDateExpression.java | 45 +
.../date/BeamSqlCurrentTimeExpression.java | 53 +
.../date/BeamSqlCurrentTimestampExpression.java | 49 +
.../date/BeamSqlDateCeilExpression.java | 55 +
.../date/BeamSqlDateFloorExpression.java | 55 +
.../date/BeamSqlDatetimePlusExpression.java | 129 +
.../operator/date/BeamSqlExtractExpression.java | 102 +
.../date/BeamSqlIntervalMultiplyExpression.java | 103 +
.../operator/date/TimeUnitUtils.java | 54 +
.../interpreter/operator/date/package-info.java | 22 +
.../operator/logical/BeamSqlAndExpression.java | 48 +
.../logical/BeamSqlLogicalExpression.java | 46 +
.../operator/logical/BeamSqlNotExpression.java | 54 +
.../operator/logical/BeamSqlOrExpression.java | 48 +
.../operator/logical/package-info.java | 22 +
.../operator/math/BeamSqlAbsExpression.java | 74 +
.../operator/math/BeamSqlAcosExpression.java | 40 +
.../operator/math/BeamSqlAsinExpression.java | 40 +
.../operator/math/BeamSqlAtan2Expression.java | 42 +
.../operator/math/BeamSqlAtanExpression.java | 40 +
.../operator/math/BeamSqlCeilExpression.java | 45 +
.../operator/math/BeamSqlCosExpression.java | 40 +
.../operator/math/BeamSqlCotExpression.java | 40 +
.../operator/math/BeamSqlDegreesExpression.java | 40 +
.../operator/math/BeamSqlExpExpression.java | 40 +
.../operator/math/BeamSqlFloorExpression.java | 45 +
.../operator/math/BeamSqlLnExpression.java | 40 +
.../operator/math/BeamSqlLogExpression.java | 40 +
.../math/BeamSqlMathBinaryExpression.java | 65 +
.../math/BeamSqlMathUnaryExpression.java | 60 +
.../operator/math/BeamSqlPiExpression.java | 43 +
.../operator/math/BeamSqlPowerExpression.java | 44 +
.../operator/math/BeamSqlRadiansExpression.java | 40 +
.../operator/math/BeamSqlRandExpression.java | 55 +
.../math/BeamSqlRandIntegerExpression.java | 59 +
.../operator/math/BeamSqlRoundExpression.java | 107 +
.../operator/math/BeamSqlSignExpression.java | 72 +
.../operator/math/BeamSqlSinExpression.java | 40 +
.../operator/math/BeamSqlTanExpression.java | 40 +
.../math/BeamSqlTruncateExpression.java | 75 +
.../interpreter/operator/math/package-info.java | 22 +
.../impl/interpreter/operator/package-info.java | 22 +
.../string/BeamSqlCharLengthExpression.java | 40 +
.../string/BeamSqlConcatExpression.java | 63 +
.../string/BeamSqlInitCapExpression.java | 56 +
.../operator/string/BeamSqlLowerExpression.java | 40 +
.../string/BeamSqlOverlayExpression.java | 77 +
.../string/BeamSqlPositionExpression.java | 73 +
.../string/BeamSqlStringUnaryExpression.java | 44 +
.../string/BeamSqlSubstringExpression.java | 83 +
.../operator/string/BeamSqlTrimExpression.java | 102 +
.../operator/string/BeamSqlUpperExpression.java | 40 +
.../operator/string/package-info.java | 22 +
.../sql/impl/interpreter/package-info.java | 22 +
.../sdk/extensions/sql/impl/package-info.java | 22 +
.../sql/impl/planner/BeamQueryPlanner.java | 168 +
.../sql/impl/planner/BeamRelDataTypeSystem.java | 40 +
.../sql/impl/planner/BeamRuleSets.java | 75 +
.../sql/impl/planner/package-info.java | 24 +
.../sql/impl/rel/BeamAggregationRel.java | 182 +
.../extensions/sql/impl/rel/BeamFilterRel.java | 69 +
.../extensions/sql/impl/rel/BeamIOSinkRel.java | 75 +
.../sql/impl/rel/BeamIOSourceRel.java | 62 +
.../sql/impl/rel/BeamIntersectRel.java | 58 +
.../extensions/sql/impl/rel/BeamJoinRel.java | 298 ++
.../sql/impl/rel/BeamLogicalConvention.java | 72 +
.../extensions/sql/impl/rel/BeamMinusRel.java | 56 +
.../extensions/sql/impl/rel/BeamProjectRel.java | 80 +
.../extensions/sql/impl/rel/BeamRelNode.java | 39 +
.../sql/impl/rel/BeamSetOperatorRelBase.java | 98 +
.../extensions/sql/impl/rel/BeamSortRel.java | 235 ++
.../sql/impl/rel/BeamSqlRelUtils.java | 72 +
.../extensions/sql/impl/rel/BeamUnionRel.java | 88 +
.../extensions/sql/impl/rel/BeamValuesRel.java | 79 +
.../extensions/sql/impl/rel/package-info.java | 23 +
.../sql/impl/rule/BeamAggregationRule.java | 162 +
.../sql/impl/rule/BeamFilterRule.java | 49 +
.../sql/impl/rule/BeamIOSinkRule.java | 81 +
.../sql/impl/rule/BeamIOSourceRule.java | 49 +
.../sql/impl/rule/BeamIntersectRule.java | 50 +
.../extensions/sql/impl/rule/BeamJoinRule.java | 53 +
.../extensions/sql/impl/rule/BeamMinusRule.java | 50 +
.../sql/impl/rule/BeamProjectRule.java | 50 +
.../extensions/sql/impl/rule/BeamSortRule.java | 51 +
.../extensions/sql/impl/rule/BeamUnionRule.java | 50 +
.../sql/impl/rule/BeamValuesRule.java | 48 +
.../extensions/sql/impl/rule/package-info.java | 23 +
.../sql/impl/schema/BaseBeamTable.java | 35 +
.../extensions/sql/impl/schema/BeamIOType.java | 28 +
.../sql/impl/schema/BeamPCollectionTable.java | 63 +
.../sql/impl/schema/BeamSqlTable.java | 54 +
.../sql/impl/schema/BeamTableUtils.java | 118 +
.../impl/schema/kafka/BeamKafkaCSVTable.java | 109 +
.../sql/impl/schema/kafka/BeamKafkaTable.java | 109 +
.../sql/impl/schema/kafka/package-info.java | 22 +
.../sql/impl/schema/package-info.java | 22 +
.../sql/impl/schema/text/BeamTextCSVTable.java | 70 +
.../schema/text/BeamTextCSVTableIOReader.java | 58 +
.../schema/text/BeamTextCSVTableIOWriter.java | 58 +
.../sql/impl/schema/text/BeamTextTable.java | 41 +
.../sql/impl/schema/text/package-info.java | 22 +
.../transform/BeamAggregationTransforms.java | 311 ++
.../impl/transform/BeamBuiltinAggregations.java | 557 +++
.../sql/impl/transform/BeamJoinTransforms.java | 161 +
.../transform/BeamSetOperatorsTransforms.java | 111 +
.../sql/impl/transform/BeamSqlFilterFn.java | 63 +
.../transform/BeamSqlOutputToConsoleFn.java | 41 +
.../sql/impl/transform/BeamSqlProjectFn.java | 72 +
.../sql/impl/transform/package-info.java | 22 +
.../extensions/sql/impl/utils/CalciteUtils.java | 113 +
.../extensions/sql/impl/utils/SqlTypeUtils.java | 63 +
.../extensions/sql/impl/utils/package-info.java | 22 +
.../beam/sdk/extensions/sql/package-info.java | 22 +
.../sql/src/main/resources/log4j.properties | 23 +
...ged.org.codehaus.commons.compiler.properties | 18 +
.../extensions/sql/BeamSqlApiSurfaceTest.java | 57 +
.../sql/BeamSqlDslAggregationTest.java | 400 ++
.../beam/sdk/extensions/sql/BeamSqlDslBase.java | 136 +
.../extensions/sql/BeamSqlDslFilterTest.java | 155 +
.../sdk/extensions/sql/BeamSqlDslJoinTest.java | 188 +
.../extensions/sql/BeamSqlDslProjectTest.java | 227 ++
.../extensions/sql/BeamSqlDslUdfUdafTest.java | 139 +
.../beam/sdk/extensions/sql/TestUtils.java | 190 +
.../impl/interpreter/BeamSqlFnExecutorTest.java | 446 +++
.../interpreter/BeamSqlFnExecutorTestBase.java | 88 +
.../operator/BeamNullExperssionTest.java | 55 +
.../operator/BeamSqlAndOrExpressionTest.java | 61 +
.../operator/BeamSqlCaseExpressionTest.java | 93 +
.../operator/BeamSqlCastExpressionTest.java | 129 +
.../operator/BeamSqlCompareExpressionTest.java | 115 +
.../operator/BeamSqlInputRefExpressionTest.java | 57 +
.../operator/BeamSqlPrimitiveTest.java | 59 +
.../BeamSqlReinterpretExpressionTest.java | 75 +
.../operator/BeamSqlUdfExpressionTest.java | 51 +
.../BeamSqlArithmeticExpressionTest.java | 237 ++
.../date/BeamSqlCurrentDateExpressionTest.java | 38 +
.../date/BeamSqlCurrentTimeExpressionTest.java | 39 +
.../BeamSqlCurrentTimestampExpressionTest.java | 39 +
.../date/BeamSqlDateCeilExpressionTest.java | 50 +
.../date/BeamSqlDateExpressionTestBase.java | 52 +
.../date/BeamSqlDateFloorExpressionTest.java | 49 +
.../date/BeamSqlDatetimePlusExpressionTest.java | 155 +
.../date/BeamSqlExtractExpressionTest.java | 103 +
.../BeamSqlIntervalMultiplyExpressionTest.java | 107 +
.../operator/date/TimeUnitUtilsTest.java | 54 +
.../logical/BeamSqlNotExpressionTest.java | 47 +
.../math/BeamSqlMathBinaryExpressionTest.java | 215 ++
.../math/BeamSqlMathUnaryExpressionTest.java | 312 ++
.../string/BeamSqlCharLengthExpressionTest.java | 44 +
.../string/BeamSqlConcatExpressionTest.java | 66 +
.../string/BeamSqlInitCapExpressionTest.java | 54 +
.../string/BeamSqlLowerExpressionTest.java | 44 +
.../string/BeamSqlOverlayExpressionTest.java | 87 +
.../string/BeamSqlPositionExpressionTest.java | 84 +
.../BeamSqlStringUnaryExpressionTest.java | 52 +
.../string/BeamSqlSubstringExpressionTest.java | 101 +
.../string/BeamSqlTrimExpressionTest.java | 103 +
.../string/BeamSqlUpperExpressionTest.java | 44 +
.../extensions/sql/impl/rel/BaseRelTest.java | 34 +
.../sql/impl/rel/BeamIntersectRelTest.java | 118 +
.../rel/BeamJoinRelBoundedVsBoundedTest.java | 203 +
.../rel/BeamJoinRelUnboundedVsBoundedTest.java | 240 ++
.../BeamJoinRelUnboundedVsUnboundedTest.java | 218 ++
.../sql/impl/rel/BeamMinusRelTest.java | 117 +
.../impl/rel/BeamSetOperatorRelBaseTest.java | 105 +
.../sql/impl/rel/BeamSortRelTest.java | 257 ++
.../sql/impl/rel/BeamUnionRelTest.java | 103 +
.../sql/impl/rel/BeamValuesRelTest.java | 104 +
.../sdk/extensions/sql/impl/rel/CheckSize.java | 41 +
.../sql/impl/schema/BeamSqlRowCoderTest.java | 77 +
.../schema/kafka/BeamKafkaCSVTableTest.java | 107 +
.../impl/schema/text/BeamTextCSVTableTest.java | 176 +
.../transform/BeamAggregationTransformTest.java | 453 +++
.../schema/transform/BeamTransformBaseTest.java | 97 +
.../sql/impl/utils/SqlTypeUtilsTest.java | 76 +
...amSqlArithmeticOperatorsIntegrationTest.java | 165 +
...mSqlBuiltinFunctionsIntegrationTestBase.java | 168 +
...amSqlComparisonOperatorsIntegrationTest.java | 329 ++
...mSqlConditionalFunctionsIntegrationTest.java | 60 +
.../BeamSqlDateFunctionsIntegrationTest.java | 125 +
.../BeamSqlLogicalFunctionsIntegrationTest.java | 43 +
.../BeamSqlMathFunctionsIntegrationTest.java | 351 ++
.../BeamSqlStringFunctionsIntegrationTest.java | 51 +
.../extensions/sql/mock/MockedBoundedTable.java | 134 +
.../sdk/extensions/sql/mock/MockedTable.java | 42 +
.../sql/mock/MockedUnboundedTable.java | 110 +
sdks/java/fn-execution/pom.xml | 82 +
.../harness/channel/ManagedChannelFactory.java | 82 +
.../harness/channel/SocketAddressFactory.java | 64 +
.../beam/harness/channel/package-info.java | 22 +
.../channel/ManagedChannelFactoryTest.java | 71 +
.../channel/SocketAddressFactoryTest.java | 56 +
.../org/apache/beam/harness/test/Consumer.java | 26 +
.../org/apache/beam/harness/test/Supplier.java | 26 +
.../apache/beam/harness/test/TestExecutors.java | 93 +
.../beam/harness/test/TestExecutorsTest.java | 175 +
.../apache/beam/harness/test/TestStreams.java | 185 +
.../beam/harness/test/TestStreamsTest.java | 109 +
sdks/java/harness/pom.xml | 128 +-
.../beam/fn/harness/BeamFnDataReadRunner.java | 15 +-
.../beam/fn/harness/BeamFnDataWriteRunner.java | 15 +-
.../beam/fn/harness/BoundedSourceRunner.java | 14 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 398 +-
.../org/apache/beam/fn/harness/FnHarness.java | 64 +-
.../org/apache/beam/fn/harness/IdGenerator.java | 33 +
.../fn/harness/PTransformRunnerFactory.java | 7 +-
.../harness/channel/ManagedChannelFactory.java | 86 -
.../harness/channel/SocketAddressFactory.java | 64 -
.../beam/fn/harness/channel/package-info.java | 22 -
.../fn/harness/control/BeamFnControlClient.java | 15 +-
.../harness/control/ProcessBundleHandler.java | 154 +-
.../fn/harness/control/RegisterHandler.java | 8 +-
.../BeamFnDataBufferingOutboundObserver.java | 8 +-
.../beam/fn/harness/data/BeamFnDataClient.java | 7 +-
.../fn/harness/data/BeamFnDataGrpcClient.java | 19 +-
.../harness/data/BeamFnDataGrpcMultiplexer.java | 24 +-
.../harness/data/BeamFnDataInboundObserver.java | 4 +-
.../beam/fn/harness/fake/FakeStepContext.java | 39 -
.../beam/fn/harness/fake/package-info.java | 22 -
.../beam/fn/harness/fn/ThrowingBiConsumer.java | 33 +
.../fn/harness/logging/BeamFnLoggingClient.java | 202 +-
.../beam/fn/harness/state/BagUserState.java | 121 +
.../fn/harness/state/BeamFnStateClient.java | 39 +
.../state/BeamFnStateGrpcClientCache.java | 173 +
.../state/LazyCachingIteratorToIterable.java | 72 +
.../harness/state/StateFetchingIterators.java | 126 +
.../beam/fn/harness/state/package-info.java | 22 +
.../harness/stream/BufferingStreamObserver.java | 16 +-
.../beam/fn/harness/stream/DataStreams.java | 73 +-
.../harness/stream/StreamObserverFactory.java | 4 +-
.../fn/harness/BeamFnDataReadRunnerTest.java | 19 +-
.../fn/harness/BeamFnDataWriteRunnerTest.java | 15 +-
.../fn/harness/BoundedSourceRunnerTest.java | 21 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 247 +-
.../apache/beam/fn/harness/FnHarnessTest.java | 25 +-
.../apache/beam/fn/harness/IdGeneratorTest.java | 40 +
.../channel/ManagedChannelFactoryTest.java | 74 -
.../channel/SocketAddressFactoryTest.java | 56 -
.../control/BeamFnControlClientTest.java | 13 +-
.../control/ProcessBundleHandlerTest.java | 160 +-
.../fn/harness/control/RegisterHandlerTest.java | 10 +-
...BeamFnDataBufferingOutboundObserverTest.java | 4 +-
.../harness/data/BeamFnDataGrpcClientTest.java | 30 +-
.../data/BeamFnDataGrpcMultiplexerTest.java | 9 +-
.../data/BeamFnDataInboundObserverTest.java | 2 +-
.../logging/BeamFnLoggingClientTest.java | 136 +-
.../beam/fn/harness/state/BagUserStateTest.java | 106 +
.../state/BeamFnStateGrpcClientCacheTest.java | 234 ++
.../fn/harness/state/FakeBeamFnStateClient.java | 110 +
.../LazyCachingIteratorToIterableTest.java | 76 +
.../state/StateFetchingIteratorsTest.java | 99 +
.../stream/BufferingStreamObserverTest.java | 12 +-
.../beam/fn/harness/stream/DataStreamsTest.java | 165 +-
.../stream/DirectStreamObserverTest.java | 8 +-
.../beam/fn/harness/test/TestExecutors.java | 85 -
.../beam/fn/harness/test/TestExecutorsTest.java | 160 -
.../beam/fn/harness/test/TestStreams.java | 162 -
.../beam/fn/harness/test/TestStreamsTest.java | 84 -
sdks/java/io/amqp/pom.xml | 24 +-
.../org/apache/beam/sdk/io/amqp/AmqpIO.java | 33 +-
.../org/apache/beam/sdk/io/amqp/AmqpIOTest.java | 112 +-
sdks/java/io/cassandra/pom.xml | 2 +-
.../beam/sdk/io/cassandra/CassandraIO.java | 64 +-
sdks/java/io/common/pom.xml | 2 +-
.../sdk/io/common/IOTestPipelineOptions.java | 6 +
.../elasticsearch-tests-2/pom.xml | 60 +
.../src/test/contrib/create_elk_container.sh | 24 +
.../sdk/io/elasticsearch/ElasticsearchIOIT.java | 123 +
.../io/elasticsearch/ElasticsearchIOTest.java | 185 +
.../elasticsearch-tests-5/pom.xml | 124 +
.../src/test/contrib/create_elk_container.sh | 24 +
.../sdk/io/elasticsearch/ElasticsearchIOIT.java | 122 +
.../io/elasticsearch/ElasticsearchIOTest.java | 185 +
.../org/elasticsearch/bootstrap/JarHell.java | 39 +
.../elasticsearch-tests-common/pom.xml | 77 +
.../elasticsearch/ElasticSearchIOTestUtils.java | 141 +
.../elasticsearch/ElasticsearchIOITCommon.java | 92 +
.../ElasticsearchIOTestCommon.java | 306 ++
sdks/java/io/elasticsearch-tests/pom.xml | 144 +
sdks/java/io/elasticsearch/pom.xml | 228 +-
.../sdk/io/elasticsearch/ElasticsearchIO.java | 448 ++-
.../beam/sdk/io/elasticsearch/package-info.java | 1 -
.../src/test/contrib/create_elk_container.sh | 24 -
.../elasticsearch/ElasticSearchIOTestUtils.java | 138 -
.../sdk/io/elasticsearch/ElasticsearchIOIT.java | 155 -
.../io/elasticsearch/ElasticsearchIOTest.java | 355 --
.../elasticsearch/ElasticsearchTestDataSet.java | 97 -
sdks/java/io/google-cloud-platform/pom.xml | 9 +-
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 473 ++-
.../sdk/io/gcp/bigquery/BigQueryHelpers.java | 28 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 755 +++-
.../io/gcp/bigquery/BigQueryQuerySource.java | 49 +-
.../sdk/io/gcp/bigquery/BigQueryServices.java | 53 +-
.../io/gcp/bigquery/BigQueryServicesImpl.java | 82 +-
.../sdk/io/gcp/bigquery/BigQuerySourceBase.java | 131 +-
.../gcp/bigquery/BigQueryTableRowIterator.java | 501 ---
.../io/gcp/bigquery/BigQueryTableSource.java | 40 +-
.../sdk/io/gcp/bigquery/CalculateSchemas.java | 78 -
.../beam/sdk/io/gcp/bigquery/CreateTables.java | 23 +-
.../io/gcp/bigquery/DynamicDestinations.java | 13 +-
.../bigquery/DynamicDestinationsHelpers.java | 27 +-
.../sdk/io/gcp/bigquery/ReifyAsIterable.java | 51 +
.../sdk/io/gcp/bigquery/SchemaAndRecord.java | 43 +
.../io/gcp/bigquery/StreamingWriteTables.java | 10 +-
.../sdk/io/gcp/bigquery/TableDestination.java | 43 +-
.../io/gcp/bigquery/TableDestinationCoder.java | 2 +
.../gcp/bigquery/TableDestinationCoderV2.java | 59 +
.../io/gcp/bigquery/WriteBundlesToFiles.java | 43 +-
.../bigquery/WriteGroupedRecordsToFiles.java | 7 +-
.../sdk/io/gcp/bigquery/WritePartition.java | 13 +-
.../beam/sdk/io/gcp/bigquery/WriteRename.java | 112 +-
.../beam/sdk/io/gcp/bigquery/WriteTables.java | 189 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 135 +-
.../io/gcp/bigtable/BigtableServiceImpl.java | 16 +-
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 111 +-
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 35 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSink.java | 6 +-
.../io/gcp/pubsub/PubsubUnboundedSource.java | 41 +-
.../sdk/io/gcp/spanner/AbstractSpannerFn.java | 71 -
.../sdk/io/gcp/spanner/CreateTransactionFn.java | 22 +-
.../io/gcp/spanner/MutationGroupEncoder.java | 660 ++++
.../io/gcp/spanner/MutationSizeEstimator.java | 48 +
.../sdk/io/gcp/spanner/NaiveSpannerReadFn.java | 19 +-
.../beam/sdk/io/gcp/spanner/OrderedCode.java | 764 ++++
.../sdk/io/gcp/spanner/ReadSpannerSchema.java | 94 +
.../sdk/io/gcp/spanner/SerializedMutation.java | 35 +
.../io/gcp/spanner/SerializedMutationCoder.java | 60 +
.../sdk/io/gcp/spanner/SpannerAccessor.java | 43 +
.../beam/sdk/io/gcp/spanner/SpannerConfig.java | 41 +-
.../beam/sdk/io/gcp/spanner/SpannerIO.java | 384 +-
.../beam/sdk/io/gcp/spanner/SpannerSchema.java | 144 +
.../sdk/io/gcp/spanner/SpannerWriteGroupFn.java | 125 -
.../beam/sdk/io/gcp/GcpApiSurfaceTest.java | 2 +
.../sdk/io/gcp/bigquery/BigQueryIOTest.java | 811 ++--
.../bigquery/BigQueryTableRowIteratorTest.java | 358 --
.../sdk/io/gcp/bigquery/BigQueryUtilTest.java | 187 -
.../io/gcp/bigquery/FakeBigQueryServices.java | 78 -
.../sdk/io/gcp/bigquery/FakeDatasetService.java | 21 +-
.../sdk/io/gcp/bigquery/FakeJobService.java | 46 +-
.../sdk/io/gcp/bigquery/TableContainer.java | 2 +
.../sdk/io/gcp/bigtable/BigtableIOTest.java | 30 +-
.../sdk/io/gcp/datastore/DatastoreV1Test.java | 167 +-
.../sdk/io/gcp/datastore/SplitQueryFnIT.java | 5 +-
.../beam/sdk/io/gcp/pubsub/PubsubIOTest.java | 55 +
.../gcp/spanner/MutationGroupEncoderTest.java | 636 ++++
.../sdk/io/gcp/spanner/OrderedCodeTest.java | 890 +++++
.../io/gcp/spanner/ReadSpannerSchemaTest.java | 134 +
.../sdk/io/gcp/spanner/SpannerIOReadTest.java | 69 -
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 453 ++-
.../sdk/io/gcp/spanner/SpannerSchemaTest.java | 61 +
.../beam/sdk/io/gcp/spanner/SpannerWriteIT.java | 5 +-
sdks/java/io/hadoop-common/pom.xml | 2 +-
sdks/java/io/hadoop-file-system/pom.xml | 2 +-
.../beam/sdk/io/hdfs/HadoopFileSystem.java | 20 +-
.../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 66 +-
sdks/java/io/hadoop/input-format/pom.xml | 7 +-
.../hadoop/inputformat/HadoopInputFormatIO.java | 48 +-
.../inputformat/HadoopInputFormatIOTest.java | 14 +-
sdks/java/io/hadoop/jdk1.8-tests/pom.xml | 2 +-
sdks/java/io/hadoop/pom.xml | 2 +-
sdks/java/io/hbase/pom.xml | 2 +-
.../io/hbase/HBaseCoderProviderRegistrar.java | 8 +-
.../org/apache/beam/sdk/io/hbase/HBaseIO.java | 1001 ++---
.../beam/sdk/io/hbase/HBaseMutationCoder.java | 27 +-
.../beam/sdk/io/hbase/HBaseResultCoder.java | 6 +-
.../beam/sdk/io/hbase/SerializableScan.java | 37 +-
.../hbase/HBaseCoderProviderRegistrarTest.java | 4 +-
.../apache/beam/sdk/io/hbase/HBaseIOTest.java | 754 ++--
.../sdk/io/hbase/HBaseMutationCoderTest.java | 4 +-
.../beam/sdk/io/hbase/HBaseResultCoderTest.java | 4 +-
.../beam/sdk/io/hbase/SerializableScanTest.java | 6 +-
sdks/java/io/hcatalog/pom.xml | 2 +-
.../apache/beam/sdk/io/hcatalog/HCatalogIO.java | 23 +-
.../beam/sdk/io/hcatalog/HCatalogIOTest.java | 24 +-
sdks/java/io/jdbc/pom.xml | 2 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIO.java | 313 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 6 -
sdks/java/io/jms/pom.xml | 2 +-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 71 +-
sdks/java/io/kafka/README.md | 36 +
sdks/java/io/kafka/pom.xml | 19 +-
.../org/apache/beam/sdk/io/kafka/KafkaIO.java | 790 +++-
.../apache/beam/sdk/io/kafka/ProducerSpEL.java | 135 +
.../apache/beam/sdk/io/kafka/KafkaIOTest.java | 233 +-
sdks/java/io/kinesis/pom.xml | 8 +-
.../beam/sdk/io/kinesis/AWSClientsProvider.java | 36 +
.../sdk/io/kinesis/GetKinesisRecordsResult.java | 8 +-
.../sdk/io/kinesis/KinesisClientProvider.java | 33 -
.../apache/beam/sdk/io/kinesis/KinesisIO.java | 125 +-
.../beam/sdk/io/kinesis/KinesisReader.java | 146 +-
.../sdk/io/kinesis/KinesisReaderCheckpoint.java | 18 -
.../beam/sdk/io/kinesis/KinesisSource.java | 40 +-
.../apache/beam/sdk/io/kinesis/RoundRobin.java | 54 -
.../beam/sdk/io/kinesis/ShardCheckpoint.java | 8 +-
.../beam/sdk/io/kinesis/ShardReadersPool.java | 162 +
.../sdk/io/kinesis/ShardRecordsIterator.java | 88 +-
.../sdk/io/kinesis/SimplifiedKinesisClient.java | 98 +-
.../io/kinesis/TransientKinesisException.java | 4 +-
.../beam/sdk/io/kinesis/AmazonKinesisMock.java | 19 +-
.../sdk/io/kinesis/KinesisMockReadTest.java | 5 +-
.../beam/sdk/io/kinesis/KinesisReaderIT.java | 5 +-
.../beam/sdk/io/kinesis/KinesisReaderTest.java | 152 +-
.../beam/sdk/io/kinesis/RoundRobinTest.java | 59 -
.../sdk/io/kinesis/ShardReadersPoolTest.java | 185 +
.../io/kinesis/ShardRecordsIteratorTest.java | 35 +-
.../io/kinesis/SimplifiedKinesisClientTest.java | 107 +
sdks/java/io/mongodb/pom.xml | 2 +-
.../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 4 -
.../apache/beam/sdk/io/mongodb/MongoDbIO.java | 62 +-
.../sdk/io/mongodb/MongoDBGridFSIOTest.java | 2 +-
.../beam/sdk/io/mongodb/MongoDbIOTest.java | 11 +
sdks/java/io/mqtt/pom.xml | 2 +-
.../org/apache/beam/sdk/io/mqtt/MqttIO.java | 49 +-
sdks/java/io/pom.xml | 12 +-
sdks/java/io/redis/pom.xml | 90 +
.../io/redis/RedisConnectionConfiguration.java | 122 +
.../org/apache/beam/sdk/io/redis/RedisIO.java | 451 +++
.../apache/beam/sdk/io/redis/package-info.java | 22 +
.../apache/beam/sdk/io/redis/RedisIOTest.java | 109 +
sdks/java/io/solr/pom.xml | 145 +
.../beam/sdk/io/solr/AuthorizedSolrClient.java | 91 +
.../beam/sdk/io/solr/JavaBinCodecCoder.java | 98 +
.../org/apache/beam/sdk/io/solr/SolrIO.java | 705 ++++
.../apache/beam/sdk/io/solr/package-info.java | 20 +
.../beam/sdk/io/solr/JavaBinCodecCoderTest.java | 81 +
.../org/apache/beam/sdk/io/solr/SolrIOTest.java | 269 ++
.../beam/sdk/io/solr/SolrIOTestUtils.java | 132 +
.../resources/cloud-minimal/conf/schema.xml | 29 +
.../resources/cloud-minimal/conf/solrconfig.xml | 48 +
sdks/java/io/tika/pom.xml | 113 +
.../apache/beam/sdk/io/tika/ParseResult.java | 144 +
.../org/apache/beam/sdk/io/tika/TikaIO.java | 284 ++
.../apache/beam/sdk/io/tika/package-info.java | 22 +
.../beam/sdk/io/tika/ParseResultTest.java | 83 +
.../org/apache/beam/sdk/io/tika/TikaIOTest.java | 149 +
.../java/io/tika/src/test/resources/damaged.pdf | 2 +
.../resources/valid/apache-beam-tika-pdf.zip | Bin 0 -> 11685 bytes
.../test/resources/valid/apache-beam-tika.odt | Bin 0 -> 12540 bytes
sdks/java/io/xml/pom.xml | 2 +-
.../java/org/apache/beam/sdk/io/xml/XmlIO.java | 375 +-
.../org/apache/beam/sdk/io/xml/XmlSink.java | 10 -
.../org/apache/beam/sdk/io/xml/XmlSource.java | 63 +-
.../org/apache/beam/sdk/io/xml/XmlSinkTest.java | 12 +-
.../apache/beam/sdk/io/xml/XmlSourceTest.java | 74 +-
sdks/java/java8tests/pom.xml | 2 +-
sdks/java/javadoc/ant.xml | 1 +
sdks/java/javadoc/pom.xml | 22 +-
.../maven-archetypes/examples-java8/pom.xml | 2 +-
.../main/resources/archetype-resources/pom.xml | 13 +
sdks/java/maven-archetypes/examples/pom.xml | 2 +-
sdks/java/maven-archetypes/pom.xml | 2 +-
sdks/java/maven-archetypes/starter/pom.xml | 2 +-
sdks/java/nexmark/README.md | 340 ++
sdks/java/nexmark/pom.xml | 265 ++
.../java/org/apache/beam/sdk/nexmark/Main.java | 303 ++
.../org/apache/beam/sdk/nexmark/Monitor.java | 78 +
.../beam/sdk/nexmark/NexmarkConfiguration.java | 721 ++++
.../beam/sdk/nexmark/NexmarkLauncher.java | 1157 ++++++
.../apache/beam/sdk/nexmark/NexmarkOptions.java | 403 ++
.../apache/beam/sdk/nexmark/NexmarkPerf.java | 207 ++
.../apache/beam/sdk/nexmark/NexmarkSuite.java | 112 +
.../apache/beam/sdk/nexmark/NexmarkUtils.java | 674 ++++
.../apache/beam/sdk/nexmark/model/Auction.java | 187 +
.../beam/sdk/nexmark/model/AuctionBid.java | 85 +
.../beam/sdk/nexmark/model/AuctionCount.java | 84 +
.../beam/sdk/nexmark/model/AuctionPrice.java | 88 +
.../org/apache/beam/sdk/nexmark/model/Bid.java | 177 +
.../beam/sdk/nexmark/model/BidsPerSession.java | 87 +
.../beam/sdk/nexmark/model/CategoryPrice.java | 97 +
.../org/apache/beam/sdk/nexmark/model/Done.java | 80 +
.../apache/beam/sdk/nexmark/model/Event.java | 171 +
.../beam/sdk/nexmark/model/IdNameReserve.java | 98 +
.../beam/sdk/nexmark/model/KnownSize.java | 26 +
.../beam/sdk/nexmark/model/NameCityStateId.java | 103 +
.../apache/beam/sdk/nexmark/model/Person.java | 163 +
.../beam/sdk/nexmark/model/SellerPrice.java | 89 +
.../beam/sdk/nexmark/model/package-info.java | 22 +
.../apache/beam/sdk/nexmark/package-info.java | 21 +
.../sdk/nexmark/queries/AbstractSimulator.java | 211 ++
.../beam/sdk/nexmark/queries/NexmarkQuery.java | 270 ++
.../sdk/nexmark/queries/NexmarkQueryModel.java | 117 +
.../apache/beam/sdk/nexmark/queries/Query0.java | 70 +
.../beam/sdk/nexmark/queries/Query0Model.java | 64 +
.../apache/beam/sdk/nexmark/queries/Query1.java | 67 +
.../beam/sdk/nexmark/queries/Query10.java | 367 ++
.../beam/sdk/nexmark/queries/Query11.java | 79 +
.../beam/sdk/nexmark/queries/Query12.java | 80 +
.../beam/sdk/nexmark/queries/Query1Model.java | 76 +
.../apache/beam/sdk/nexmark/queries/Query2.java | 79 +
.../beam/sdk/nexmark/queries/Query2Model.java | 80 +
.../apache/beam/sdk/nexmark/queries/Query3.java | 301 ++
.../beam/sdk/nexmark/queries/Query3Model.java | 124 +
.../apache/beam/sdk/nexmark/queries/Query4.java | 116 +
.../beam/sdk/nexmark/queries/Query4Model.java | 186 +
.../apache/beam/sdk/nexmark/queries/Query5.java | 138 +
.../beam/sdk/nexmark/queries/Query5Model.java | 176 +
.../apache/beam/sdk/nexmark/queries/Query6.java | 155 +
.../beam/sdk/nexmark/queries/Query6Model.java | 133 +
.../apache/beam/sdk/nexmark/queries/Query7.java | 90 +
.../beam/sdk/nexmark/queries/Query7Model.java | 130 +
.../apache/beam/sdk/nexmark/queries/Query8.java | 98 +
.../beam/sdk/nexmark/queries/Query8Model.java | 148 +
.../apache/beam/sdk/nexmark/queries/Query9.java | 44 +
.../beam/sdk/nexmark/queries/Query9Model.java | 44 +
.../beam/sdk/nexmark/queries/WinningBids.java | 418 +++
.../nexmark/queries/WinningBidsSimulator.java | 206 ++
.../beam/sdk/nexmark/queries/package-info.java | 22 +
.../sdk/nexmark/sources/BoundedEventSource.java | 190 +
.../beam/sdk/nexmark/sources/Generator.java | 609 +++
.../sdk/nexmark/sources/GeneratorConfig.java | 298 ++
.../nexmark/sources/UnboundedEventSource.java | 329 ++
.../beam/sdk/nexmark/sources/package-info.java | 22 +
.../nexmark/src/main/resources/log4j.properties | 55 +
.../beam/sdk/nexmark/queries/QueryTest.java | 185 +
.../nexmark/sources/BoundedEventSourceTest.java | 70 +
.../beam/sdk/nexmark/sources/GeneratorTest.java | 110 +
.../sources/UnboundedEventSourceTest.java | 105 +
sdks/java/pom.xml | 5 +-
sdks/pom.xml | 6 +-
sdks/python/apache_beam/__init__.py | 47 +-
sdks/python/apache_beam/coders/coder_impl.py | 22 +-
sdks/python/apache_beam/coders/coders.py | 118 +-
sdks/python/apache_beam/coders/coders_test.py | 2 +-
.../apache_beam/coders/coders_test_common.py | 17 +-
.../apache_beam/coders/observable_test.py | 1 -
.../coders/proto2_coder_test_messages_pb2.py | 6 +-
.../apache_beam/coders/standard_coders_test.py | 11 +-
sdks/python/apache_beam/coders/stream_test.py | 1 -
sdks/python/apache_beam/coders/typecoders.py | 1 -
.../examples/complete/autocomplete.py | 6 +-
.../examples/complete/autocomplete_test.py | 2 +-
.../examples/complete/estimate_pi.py | 5 +-
.../examples/complete/estimate_pi_test.py | 2 +-
.../examples/complete/game/game_stats.py | 393 ++
.../examples/complete/game/game_stats_test.py | 81 +
.../examples/complete/game/hourly_team_score.py | 247 +-
.../examples/complete/game/leader_board.py | 349 ++
.../examples/complete/game/leader_board_test.py | 69 +
.../examples/complete/game/user_score.py | 186 +-
.../complete/juliaset/juliaset/juliaset.py | 8 +-
.../complete/juliaset/juliaset/juliaset_test.py | 1 -
.../examples/complete/juliaset/juliaset_main.py | 2 -
.../examples/complete/juliaset/setup.py | 7 +-
.../apache_beam/examples/complete/tfidf.py | 17 +-
.../apache_beam/examples/complete/tfidf_test.py | 7 +-
.../examples/complete/top_wikipedia_sessions.py | 5 +-
.../complete/top_wikipedia_sessions_test.py | 1 -
.../examples/cookbook/bigquery_side_input.py | 5 +-
.../examples/cookbook/bigquery_tornadoes.py | 2 +-
.../cookbook/bigquery_tornadoes_it_test.py | 2 +-
.../examples/cookbook/custom_ptransform.py | 2 +-
.../examples/cookbook/datastore_wordcount.py | 39 +-
.../examples/cookbook/group_with_coder.py | 6 +-
.../examples/cookbook/group_with_coder_test.py | 1 -
.../examples/cookbook/mergecontacts.py | 36 +-
.../examples/cookbook/multiple_output_pardo.py | 14 +-
.../apache_beam/examples/snippets/snippets.py | 82 +-
.../examples/snippets/snippets_test.py | 70 +-
.../apache_beam/examples/streaming_wordcount.py | 9 +-
.../apache_beam/examples/windowed_wordcount.py | 7 +-
sdks/python/apache_beam/examples/wordcount.py | 12 +-
.../apache_beam/examples/wordcount_debugging.py | 12 +-
.../apache_beam/examples/wordcount_fnapi.py | 151 +
.../apache_beam/examples/wordcount_it_test.py | 16 +
.../apache_beam/examples/wordcount_minimal.py | 6 +-
sdks/python/apache_beam/internal/gcp/auth.py | 1 -
.../apache_beam/internal/gcp/json_value.py | 46 +-
.../apache_beam/internal/gcp/json_value_test.py | 3 +-
sdks/python/apache_beam/internal/pickler.py | 2 +-
sdks/python/apache_beam/internal/util.py | 6 +-
sdks/python/apache_beam/io/avroio.py | 164 +-
sdks/python/apache_beam/io/avroio_test.py | 54 +-
.../python/apache_beam/io/concat_source_test.py | 1 -
sdks/python/apache_beam/io/filebasedsink.py | 22 +-
.../python/apache_beam/io/filebasedsink_test.py | 3 +-
sdks/python/apache_beam/io/filebasedsource.py | 234 +-
.../apache_beam/io/filebasedsource_test.py | 8 +-
sdks/python/apache_beam/io/filesystem.py | 31 +-
sdks/python/apache_beam/io/filesystem_test.py | 5 +-
sdks/python/apache_beam/io/filesystems.py | 1 -
sdks/python/apache_beam/io/filesystems_test.py | 4 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 264 +-
sdks/python/apache_beam/io/gcp/bigquery_test.py | 4 +-
.../io/gcp/datastore/v1/adaptive_throttler.py | 94 +
.../gcp/datastore/v1/adaptive_throttler_test.py | 95 +
.../io/gcp/datastore/v1/datastoreio.py | 55 +-
.../io/gcp/datastore/v1/datastoreio_test.py | 11 +-
.../apache_beam/io/gcp/datastore/v1/helper.py | 31 +-
.../io/gcp/datastore/v1/helper_test.py | 7 +-
.../apache_beam/io/gcp/gcsfilesystem_test.py | 1 +
sdks/python/apache_beam/io/gcp/gcsio.py | 17 +-
.../clients/bigquery/bigquery_v2_messages.py | 3 -
.../clients/storage/storage_v1_messages.py | 3 -
sdks/python/apache_beam/io/gcp/pubsub.py | 5 +-
sdks/python/apache_beam/io/gcp/pubsub_test.py | 5 +-
.../io/gcp/tests/bigquery_matcher_test.py | 3 +-
sdks/python/apache_beam/io/gcp/tests/utils.py | 8 +-
.../apache_beam/io/gcp/tests/utils_test.py | 70 +-
sdks/python/apache_beam/io/iobase.py | 83 +-
.../apache_beam/io/localfilesystem_test.py | 4 +-
sdks/python/apache_beam/io/range_trackers.py | 54 +-
.../apache_beam/io/range_trackers_test.py | 37 +
sdks/python/apache_beam/io/source_test_utils.py | 92 +-
.../apache_beam/io/source_test_utils_test.py | 2 +-
sdks/python/apache_beam/io/sources_test.py | 1 -
sdks/python/apache_beam/io/textio.py | 178 +-
sdks/python/apache_beam/io/textio_test.py | 109 +-
sdks/python/apache_beam/io/tfrecordio.py | 5 +-
sdks/python/apache_beam/io/tfrecordio_test.py | 8 +-
sdks/python/apache_beam/metrics/cells_test.py | 2 +-
sdks/python/apache_beam/metrics/execution.py | 5 +-
.../apache_beam/metrics/execution_test.py | 8 +-
sdks/python/apache_beam/metrics/metric.py | 3 +-
sdks/python/apache_beam/metrics/metric_test.py | 18 +-
sdks/python/apache_beam/metrics/metricbase.py | 4 +
.../apache_beam/options/pipeline_options.py | 32 +-
.../options/pipeline_options_test.py | 7 +-
.../options/pipeline_options_validator_test.py | 3 +-
.../apache_beam/options/value_provider.py | 1 -
sdks/python/apache_beam/pipeline.py | 136 +-
sdks/python/apache_beam/pipeline_test.py | 92 +-
sdks/python/apache_beam/pvalue.py | 111 +-
sdks/python/apache_beam/pvalue_test.py | 8 +
sdks/python/apache_beam/runners/common.py | 6 +-
sdks/python/apache_beam/runners/common_test.py | 2 +-
.../runners/dataflow/dataflow_metrics.py | 2 +-
.../runners/dataflow/dataflow_runner.py | 122 +-
.../runners/dataflow/dataflow_runner_test.py | 89 +-
.../runners/dataflow/internal/apiclient.py | 106 +-
.../runners/dataflow/internal/apiclient_test.py | 112 +-
.../clients/dataflow/dataflow_v1b3_client.py | 259 +-
.../clients/dataflow/dataflow_v1b3_messages.py | 92 +-
.../clients/dataflow/message_matchers.py | 1 -
.../clients/dataflow/message_matchers_test.py | 3 +-
.../runners/dataflow/internal/dependency.py | 58 +-
.../dataflow/internal/dependency_test.py | 10 +-
.../runners/dataflow/internal/names.py | 2 +
.../runners/dataflow/native_io/iobase_test.py | 22 +-
.../dataflow/native_io/streaming_create.py | 2 +-
.../runners/dataflow/template_runner_test.py | 5 +-
.../runners/dataflow/test_dataflow_runner.py | 5 +-
.../consumer_tracking_pipeline_visitor_test.py | 2 +-
.../runners/direct/direct_metrics.py | 2 +-
.../runners/direct/direct_metrics_test.py | 8 +-
.../apache_beam/runners/direct/direct_runner.py | 19 +-
.../runners/direct/direct_runner_test.py | 41 +
.../runners/direct/evaluation_context.py | 31 +-
.../apache_beam/runners/direct/executor.py | 127 +-
.../runners/direct/helper_transforms.py | 2 +-
.../runners/direct/transform_evaluator.py | 67 +-
sdks/python/apache_beam/runners/direct/util.py | 7 +-
.../runners/experimental/__init__.py | 16 +
.../experimental/python_rpc_direct/__init__.py | 22 +
.../python_rpc_direct_runner.py | 110 +
.../experimental/python_rpc_direct/server.py | 111 +
sdks/python/apache_beam/runners/job/__init__.py | 16 +
sdks/python/apache_beam/runners/job/manager.py | 52 +
sdks/python/apache_beam/runners/job/utils.py | 32 +
.../apache_beam/runners/pipeline_context.py | 2 +-
.../runners/portability/fn_api_runner.py | 393 +-
.../runners/portability/fn_api_runner_test.py | 115 +-
.../portability/maptask_executor_runner.py | 19 +-
.../portability/maptask_executor_runner_test.py | 12 +-
.../portability/universal_local_runner.py | 409 ++
.../portability/universal_local_runner_main.py | 44 +
.../portability/universal_local_runner_test.py | 85 +
sdks/python/apache_beam/runners/runner.py | 61 +-
sdks/python/apache_beam/runners/runner_test.py | 2 +-
.../runners/worker/bundle_processor.py | 235 +-
.../apache_beam/runners/worker/data_plane.py | 35 +-
.../runners/worker/data_plane_test.py | 7 +-
.../apache_beam/runners/worker/log_handler.py | 17 +-
.../runners/worker/log_handler_test.py | 20 +-
.../apache_beam/runners/worker/opcounters.py | 10 +-
.../runners/worker/opcounters_test.py | 1 -
.../apache_beam/runners/worker/operations.pxd | 4 +-
.../apache_beam/runners/worker/operations.py | 84 +-
.../apache_beam/runners/worker/sdk_worker.py | 206 +-
.../runners/worker/sdk_worker_main.py | 30 +-
.../runners/worker/sdk_worker_test.py | 11 +-
.../apache_beam/runners/worker/statesampler.pyx | 68 +-
.../runners/worker/statesampler_fake.py | 17 +-
.../runners/worker/statesampler_test.py | 5 +-
.../apache_beam/testing/pipeline_verifiers.py | 1 -
.../testing/pipeline_verifiers_test.py | 5 +-
.../python/apache_beam/testing/test_pipeline.py | 54 +-
.../apache_beam/testing/test_pipeline_test.py | 4 +-
sdks/python/apache_beam/testing/test_stream.py | 1 -
.../apache_beam/testing/test_stream_test.py | 3 +-
sdks/python/apache_beam/testing/test_utils.py | 23 +-
.../apache_beam/testing/test_utils_test.py | 62 +
sdks/python/apache_beam/testing/util.py | 5 +-
sdks/python/apache_beam/testing/util_test.py | 4 +-
sdks/python/apache_beam/transforms/combiners.py | 9 +-
.../apache_beam/transforms/combiners_test.py | 5 +-
sdks/python/apache_beam/transforms/core.py | 397 +-
.../apache_beam/transforms/create_test.py | 6 +-
sdks/python/apache_beam/transforms/display.py | 91 +-
.../apache_beam/transforms/display_test.py | 6 +-
.../python/apache_beam/transforms/ptransform.py | 147 +-
.../apache_beam/transforms/ptransform_test.py | 78 +-
.../apache_beam/transforms/sideinputs_test.py | 5 +-
sdks/python/apache_beam/transforms/timeutil.py | 1 -
sdks/python/apache_beam/transforms/trigger.py | 32 +-
.../apache_beam/transforms/trigger_test.py | 19 +-
sdks/python/apache_beam/transforms/util.py | 273 +-
sdks/python/apache_beam/transforms/util_test.py | 108 +
sdks/python/apache_beam/transforms/window.py | 9 +-
.../apache_beam/transforms/window_test.py | 17 +-
.../transforms/write_ptransform_test.py | 4 +-
sdks/python/apache_beam/typehints/decorators.py | 129 +-
.../typehints/native_type_compatibility.py | 166 +
.../typehints/native_type_compatibility_test.py | 92 +
sdks/python/apache_beam/typehints/opcodes.py | 19 +-
.../apache_beam/typehints/trivial_inference.py | 80 +-
.../typehints/trivial_inference_test.py | 9 +-
sdks/python/apache_beam/typehints/typecheck.py | 6 +-
.../typehints/typed_pipeline_test.py | 33 +-
sdks/python/apache_beam/typehints/typehints.py | 79 +-
.../apache_beam/typehints/typehints_test.py | 3 +-
.../apache_beam/utils/annotations_test.py | 1 +
sdks/python/apache_beam/utils/counters.py | 107 +-
sdks/python/apache_beam/utils/counters_test.py | 78 +
sdks/python/apache_beam/utils/processes_test.py | 1 -
sdks/python/apache_beam/utils/proto_utils.py | 11 +
sdks/python/apache_beam/utils/retry.py | 9 +-
sdks/python/apache_beam/utils/retry_test.py | 5 +-
sdks/python/apache_beam/utils/urns.py | 22 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/container/Dockerfile | 27 +
sdks/python/container/boot.go | 123 +
sdks/python/container/pom.xml | 154 +
sdks/python/gen_protos.py | 30 +-
sdks/python/generate_pydoc.sh | 134 +-
sdks/python/pom.xml | 9 +-
sdks/python/run_postcommit.sh | 20 -
sdks/python/run_pylint.sh | 45 +-
sdks/python/run_validatesrunner.sh | 71 +
sdks/python/setup.py | 10 +-
sdks/python/tox.ini | 21 +-
1247 files changed, 92590 insertions(+), 17747 deletions(-)
----------------------------------------------------------------------