You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ke...@apache.org on 2019/01/15 20:23:19 UTC
[beam] branch release-2.10.0 updated (6cd27b0 -> 343e662)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch release-2.10.0
in repository https://gitbox.apache.org/repos/asf/beam.git.
discard 6cd27b0 [BEAM-6418] Execute Flink tests serially to avoid memory issues
omit de94544 [BEAM-6294] Ensure input and output coders are equal for reshuffle transforms.
omit 629eaf9 Merge pull request #7464: [BEAM-6382] Cherry pick pr #7443 into 2.10.0 release branch
omit 57af122 [BEAM-6382]: Fix styling issues
omit 19bb897 [BEAM-6248] Add Flink v1.7 build target to Flink Runner (#7300)
omit ae38b89 [BEAM-6326] Fix portable stateful processing with side input
omit 5e398fe [BEAM-6382] SamzaRunner: add an option to read configs using a user-defined factory
omit 93a4cb5 Merge pull request #7419: [BEAM-6337] Cherry-pick #7417 to release-2.10.0: Fix JdbcDriver breaking DriverManager#getConnection
omit 500b25f [BEAM-6367] Fix JdbcDriver breaking DriverManager#getConnection
add 3b8abca Upgrade vendored gRPC artifact version to 0.2
add 15aa88d Merge pull request #7328: [BEAM-6056] Upgrade vendored gRPC artifact version to 0.2
add 14781c7 [BEAM-6056] Source vendored grpc dependency from Maven central
add a25b64d Merge pull request #7388: [BEAM-6056] Source vendored grpc dependency from Maven central
add 095870f Python 3 port io.range_trackers
add 359ddb9 Add io.restriction_trackers_test to Python 3 test suite
add bca5c60 Merge pull request #7358 from RobbeSneyders/trackers
add 5ce0933 Updates release validating to run LeaderBoard example using Dataflow Streaming Engine
add 5dd597e Merge pull request #7365: [BEAM-6249] Adds an Streaming Engine based test to release validation
add 5cdf3a7 [BEAM-5315] Python 3 port io.source* and io.concat_source* modules (#7383)
add 4b039e4 [BEAM-5315] Python 3 port io.filebased_* modules (#7386)
add fc482f1 [BEAM-5959] Add performance testing for writing many files
add 41dd6e1 Merge pull request #7266 from udim/cmek-perf
add a24b1af Move org.apache.beam.runners.samza.util.Base64Serializer to org.apache.beam.runners.core.serialization.Base64Serializer to be used by other runners
add 3b8ae00 Fix visibility of deserialize method
add 0783779 Add missing package-info
add 4660895 Merge pull request #7384 from echauchot/Base64Serializer
add c4590a0 split SerializablePipelineOptions into serialization utils and instance class.
add 5130bcb Merge pull request #7385 from echauchot/exposeSerializationSerializablePipelineOptions
add a404cee Add paddedcell fix to spotlessJava rules.
add c148c35 Merge pull request #7390: [BEAM-6339] Add paddedcell fix to spotlessJava rules.
add c028ebc Upgrade html-proofer and dependencies to latest
add 07c279a Remove broken links to datatorrent.com
add b09e721 Fix pydoc link to GoogleCloudOptions
add fd5e321 Remove broken link to atrato.io
add a79ef89 Fix link to internal anchor
add 5466ac0 Remove stale exclusions from HTML link checker.
add a2986cc Merge pull request #7393: [BEAM-5662] Clean up website html-proofer config
add b02f79f Disable UsesMetricsPusher tests for direct-runner
add f74c979 Fix SplittableDoFnTest#testBoundedness
add 459e730 [BEAM-6352] Ignore tests using Watch PTransform
add 26c73ef [BEAM-6353] Fix TFRecordIOTest
add 92a6c23 [BEAM-6354] Add timeout and ignore hanging tests
add 55ffd97 Add :beam-runners-direct-java:needsRunnerTests to javaPreCommit
add c591727 Merge pull request #7374: Add :beam-runners-direct-java:needsRunnerTests to javaPreCommit
add 3948595 [BEAM-5959] Reorder methods according to convention
add 5716dba Merge pull request #7403 from udim/cmek-perf
add 5212b71 [BEAM-6030] Split metrics related options out of PipelineOptions
add 185cb1a [BEAM-6030] Add Experimental label on MetricsOptions
add bd80118 Merge pull request #7400 from echauchot/BEAM-6030-metrics-sinks-pipelineOptions
add ea0b8d5 Upgrade Gradle to 4.10.3
add 20abb3e Merge pull request #7401: Upgrade Gradle to 4.10.3
add 8462a8b BEAM-4110 fix and reenable ResourceIdTester.runResourceIdBattery
add 3670b18 BEAM-4110 fixed deprecation warnings and cleanup
add e832e08 BEAM-4143 reenable sickbayed test, cleanup
add b4d2d8e Merge pull request #7412: [BEAM-4143] GcsResourceIdTest has had a masked failure
add a1d938e [BEAM-6367] Fix JdbcDriver breaking DriverManager#getConnection
add 4b5b1a0 Merge pull request #7417 from kanterov/kanterov_fix_beam_driver
add 1c76927 BQ Geography support for Java SDK
add 3739683 Merge pull request #7389 from pabloem/bq-jeo
add d19585a [BEAM-4030] Add compact attribute to combiners (#7405)
add 8a7f971 [BEAM-6362] remove --info from gradle invocations
add f22d12d Reduce days to keep Jenkins job logs to 14
add a0bc8a4 Merge pull request #7410: Reduce days to keep Jenkins job logs to 14
add 1315bbb Improve BigQuery test utils and BigQueryToTableIT
add 029712c Fix Andrew's comments
add a896bbc Merge pull request #7378 from boyuanzz/fix_bq
add 574526f Plumb the contexts though the gcsx library.
add 6c024c6 Merge pull request #7292 from bramp/BEAM-6155
add abe3220 [BEAM-6009] Modify Github template and README to include Flink PVR for batch/streaming (#7370)
add 0f892b5 [BEAM-5985] Create jenkins jobs to run the load tests for Java SDK
add cdf3f1d [BEAM-5985] Introduced Small Load test suite.
add 7604c4a [BEAM-5985] Extracted common part for Nexmark and LoadTests in Jenkins configuration. Some fixes.
add e565087 Merge pull request #7184: [BEAM-5985] Create jenkins jobs to run the load tests for Java SDK
add 5301992 [website] Add Integration page with information on Kettle
add a95e964 Merge pull request #7308: [website] Add Integration page with information on Kettle
add dc25f5e [BEAM-6188] Move SyntheticSourceOptions to upper level
add ba62514 [BEAM-6188] Separate bundle splitting logic from the IO
add eaff6c6 [BEAM-6188] Separate source delay logic from the IO
add cf19855 [BEAM-6188] Move SyntheticBoundedSource to upper level
add 5def687 [BEAM-6188] Add UnboundedSyntheticSource for streaming tests
add 2c31821 [BEAM-6188] Enable using UnboundedSyntheticSource in load tests
add a61a79c [BEAM-6188] Remove unused code
add 9460ad5 [BEAM-6188] Make watermark go to +inf if there are no more records
add 032c3ed [BEAM-6188] Remove deduping related code (deduping is not required)
add ac18fac Merge pull request #7226: [BEAM-6188] Unbouded synthetic source
add ebb9452 ClickHouseIO: setup gradle module
add 96fed9d ClickHouseIO: minimal code for inserts
add b4c892f ClickHouseIO: parsing of table schema
add 3bdd48a ClickHouseIO: implement writing
add 054b88c Merge pull request #7006: [BEAM-5964] Add ClickHouseIO.Write
add d802843 BEAM-6378 - Updating Tika
add 28353e8 Merge pull request #7387: [BEAM-6378] Updating Tika
add f43ea10 Fix Portable VR configuration
add 24f83f0 Merge pull request #7407: Fix Dataflow Java Portable VR configuration
add 717ddba Change warning logic to only consider work items without an ID to be invalid. Not the absence of a work item.
add a4e3083 Merge pull request #7421: Change warning logic to only consider work items without an ID to be invalid. Not the absence of a work item.
add 185eca3 Add some Go examples to the code snippets. (#7084)
add 7bf0c75 Use millisecond resolution for output file names to ensure uniqueness
add d02b859 Merge pull request #7424 from robinyqiu/master
add 8da36cf [BEAM-6363] Upgrade Gearpump to 0.9.0
add 540a97d [BEAM-6357] Retry Datastore writes on ABORTED
add f48c21d Merge pull request #7406 from udim/datastore-aborted
add bf2853e remove redundant assignments
add ef7463d Merge pull request #7431: remove redundant assignments
add 109eeb8 Merge pull request #7303: Add Learning Resources page
add bcccea1 [BEAM-6021] Serialize more core classes when using kryo's BeamSparkRunnerRegistrator
add e38f0b2 Merge pull request #6998: [BEAM-6021] Serialize more core classes when using kryo's BeamSparkRunnerRegistrator
add 5e7d57a [BEAM-5798] Add support to write to multiple topics with KafkaIO
add e124ace Merge pull request #7371: [BEAM-5798] Add support to write to multiple topics with KafkaIO
add c379622 Merge pull request #7289: [BEAM-6240] Add a library of schema annotations for POJO and JavaBeans
add e4d4047 Clean up stream timeout handling for gRPC windmill streams to prevent incorrect overriding in tests
add 03b66e7 Implement schemas for AutoValue classes.
add f597622 Merge pull request #7334: [BEAM-6293] Infer schemas from AutoValue classes.
add 27c7f0d [BEAM-6347] Add page for developing I/O connectors for Java
add b06f635 Merge pull request #7397: [BEAM-6347] Add website page for developing I/O connectors for Java
add 5a934ff Python 3 port textio
add 33ac2f6 Copy helpers into textio_test
add c7935b1 Merge pull request #7394 from RobbeSneyders/textio
add f77ddea Make combine fn fanout optional.
add d37db5e Merge pull request #7444 Make combine fn fanout optional.
add 8ef4a0d Fix IntelliJ issues with ClickHouseIO
add 8bfa24d Merge pull request #7442: Fix IntelliJ issues with ClickHouseIO
add 69e099b [BEAM-6388] Fix flaky AtomicInsertTest#testAtomicInsert
add 16a2e8d Merge pull request #7440: [BEAM-6388] Fix flaky AtomicInsertTest#testAtomicInsert
add 93d0882 [BEAM-6348] Add ValueProvider support for Statement in JdbcIO.write()
add 54341bd [BEAM-6348] Reuse ValueProvider methods on method invocation on JdbcIO
add 3b10890 Merge pull request #7434: [BEAM-6348] Add ValueProvider support for Statement in JdbcIO.write()
add 1ae9850 [BEAM-6097] NemoRunner
add 88acc8e Merge pull request #7236: [BEAM-6097] Add information about NemoRunner
add 02f3a5e [BEAM-6346] Inspect the docker container state
add dffe2c1 Merge pull request #7395: [BEAM-6346] Inspect the docker container state
add 1607839 [BEAM-6382] SamzaRunner: add an option to read configs using a user-defined factory
add 7c45d2c Update javadoc
add 451a61e Address pr feedback
add bacec1f Merge pull request #7443: [BEAM-6382] SamzaRunner: add an option to read configs using a user-defined factory
add 90a33d2 [BEAM-6349] & [BEAM-6368] Build worker and use it when running load tests on Dataflow
add 3907e01 Merge pull request #7435: [BEAM-6349] & [BEAM-6368] Build worker and use it when running loadtests on Dataflow
add cdca82a Disable memory usage test since it fails on some platforms with urllib3 v1.24.1
add 3b2bd96 Merge pull request #7448 from kadircet/master
add dc6a03b Fix wordcount examples in Python 3
add ca87e7c Merge pull request #7447 from RobbeSneyders/wordcount
add b1a14b7 Revert "Treat VarInt encoding as a Beam primitive encoding in Dataflow runner (#7351)"
add a4a7fcf Merge pull request #7452 from boyuanzz/revert_cloudobj
add 44850a2 [BEAM-6326] Relax test assumption for PAssertTest#testWindowedIsEqualTo
add 0e6c92a Merge pull request #7432: [BEAM-6326] Relax test assumption for PAssertTest#testWindowedIsEqualTo
add 4d0ceb0 [BEAM-6397] Added PubsubMessge attribute with map of synthetic input data
add f2e6474 [BEAM-6397] Fix NoClassDefFoundError by changing GCP dependency in gradle.
add f62f490 Merge pull request #7449: [BEAM-6397] add map attribute in SyntheticDataPubSubPublisher
add aad58d1 [BEAM-6405] Reduce parallelism of PortableValidatesRunner tests
add 14426af Merge pull request #7459: [BEAM-6405] Reduce parallelism of PortableValidatesRunner tests
add b9e51f6 [BEAM-6326] Fix portable stateful processing with side input
add 49d5cf5 Merge pull request #7453: [BEAM-6326] Fix portable stateful processing with side input
add ee57028 [BEAM-6294] Ensure input and output coders are equal for reshuffle transforms.
add 336fe0c Merge pull request #7457: [BEAM-6294] Ensure input and output coders are equal for reshuffle transforms.
add 07d0a95 Fix typo in comment.
add 8489288 Merge pull request #7451: Fix typo in comment.
add c739ca5 [BEAM-6248] Add Flink v1.7 build target to Flink Runner (#7300)
add 547e3de [BEAM-3612] Generate type assertion shims for beam.
add 627f2bc Merge pull request #7361 from lostluck/shims
add da4a1d4 Refresh worker container version to master-20190110
add 077ab5a Merge pull request #7460 from HuangLED/container-build
add ec0eb04 [BEAM-6332] Lazy serialization of aggregation results in Spark runner.
add 3a53132 Merge pull request #7398: [BEAM-6332] Lazy serialization of aggregation results in Spark runner
add 73bf5e8 Extract number of worker threads from pipeline options.
add aa9bdec Add and use loopback mode for Beam Python Flink testing matrix.
add ecdb623 Merge pull request #7483: Use loopback mode for portable flinkValidatesRunner
add cc2630f Add workaround for buggy behavior in Python 3 email_generator class
add 8818f51 Merge pull request #7438 from charlesccychen/fix-py3-corruption
add b11129e [BEAM-5918] Fix casting of non-numeric types
add 7b3a513 Remove the upper bound for pytz
add cee2f61 Merge pull request #7487 from apache/aaltay-patch-2
add ebfb0f5 [BEAM-6248] Disable tests for Flink 1.7 build target
add 6b78709 Merge pull request #7489: [BEAM-6248] Disable tests for Flink 1.7 build target
add aebe89e Add a test-only dependency on tenacity, a general-purpose retry library. See: https://pypi.org/project/tenacity/
add 7d15def Make statesampler_test easier to configure, set the fastest state to last 1 second, retry once on failure to reduce flakiness
add 266eefd Give the test 3 attempts to pass.
add d25d800 Merge pull request #7492 from tvalentyn/deflake_statesampler_test
add 8ee4290 [BEAM-6418] Avoid Jenkins memory problems for Flink build targets
add dbde3e5 Merge pull request #7496: [BEAM-6418] Avoid Jenkins memory problems for Flink build targets
add c25d528 [BEAM-6421] Fix wrong command in the contribution guide
add 003d9b1 Merge pull request #7495: [BEAM-6421] Fix wrong command in the contribution guide
add 247e93e Port shaded Guava to vendored Guava
add b953645 Merge pull request #7494: [BEAM-3608] Port shaded Guava to vendored Guava
add 213f6c9 [BEAM-6184] Enforce javadoc check on non-trivial public methods (#7454)
add 452e0d1 [BEAM-6350] Reuse same PCollectionView when created in translators
add 2b90576 [BEAM-6350] AbstractJoinTranslator extended in order to prevent code muliplication. PCollectionViews are now stored in BroadcastHashJoinTranslator. Key extractor is taken into consideration when looking for the same views.
add 1ba3efb [BEAM-6350] Test added.
add 9780341 Merge pull request #7399: [BEAM-6350] Reuse PCollectionView when created in translators
add d10a971 [BEAM-6405] Let PortableValidatesRunner tests run in EMBEDDED environment
add 3db71dd Merge pull request #7461: [BEAM-6405] Let PortableValidatesRunner tests run in EMBEDDED environment
add 4019e86 Initialize project in @BeforeClass method rather than in instance initialization.
add 115473c Merge pull request #7493: Initialize project in @BeforeClass method
add 30df0ab Made SampleFixedSizeGlobally PTransform classes
add fe22868 Merge pull request #7498 from brills/samplen
add 258bf94 BEAM-6234 -> Make failOnCheckpointingErrors setting available in FlinkPipelineOptions
add c38d84c Merge pull request #7284: [BEAM-6234] Add failOnCheckpointingErrors setting in FlinkPipelineOptions
add 73cb1b9 [SQL] Refactor JdbcDriver
add f8a5dd6 Merge pull request #7488 from akedin/refactor-jdbc-driver
add ddf4cb4 fix possible out of bound exception
add 1945506 Merge pull request #7429: [BEAM-6391] fix possible out of bound exception
add 6cf9fba add missing raise keywords
add 1a898c3 Merge pull request #7428: [BEAM-6391] add missing raise keywords
add e5eef8b Update 2017-08-04-splittable-do-fn.md
add 25bd43e Merge pull request #7305: Update 2017-08-04-splittable-do-fn.md
add ffec471 fix always true conditions
add fff053b Merge pull request #7430: [BEAM-6391] fix always true conditions
add d05da8c Pin spotless and googleJavaFormat to latest; apply globally
add a443030 Fix typo
add 2f9d711 Merge pull request #7501 from tvalentyn/patch-32
add 2bcf9fd [BEAM-5319] Python 3 port runners module (#7445)
add 761240f [BEAM-6430] Fix EXCEPT.
add d7a0a3f [BEAM-6427]INTERSECT ALL is not compatible with SQL standard.
add 6480962 BEAM-6425 - Replace SSLContext.getInstance("SSL")
add 317a8b5 Merge pull request #7499: [BEAM-6425] - Replace SSLContext.getInstance("SSL")
new 343e662 Prepare for 2.10.0 release
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (6cd27b0)
\
N -- N -- N refs/heads/release-2.10.0 (343e662)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/PULL_REQUEST_TEMPLATE.md | 2 +-
.test-infra/jenkins/CommonJobProperties.groovy | 4 +-
.../jenkins/CommonTestProperties.groovy | 29 +-
.test-infra/jenkins/LoadTestsBuilder.groovy | 54 +++
.test-infra/jenkins/NexmarkBuilder.groovy | 22 +-
.test-infra/jenkins/job_LoadTests_Java.groovy | 74 +++
.../job_PerformanceTests_FileBasedIO_IT.groovy | 16 +
...job_PerformanceTests_FileBasedIO_IT_HDFS.groovy | 13 +
.../job_PostCommit_Java_Nexmark_Dataflow.groovy | 4 +-
.../job_PostCommit_Java_Nexmark_Direct.groovy | 4 +-
.../job_PostCommit_Java_Nexmark_Flink.groovy | 4 +-
.../job_PostCommit_Java_Nexmark_Spark.groovy | 3 +-
..._ValidatesRunner_PortabilityApi_Dataflow.groovy | 2 +-
.../job_beam_PerformanceTests_Analysis.groovy | 2 +
README.md | 3 +-
build.gradle | 3 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 34 +-
examples/java/build.gradle | 2 +-
.../apache/beam/examples/common/ExampleUtils.java | 8 +-
.../examples/common/WriteOneFilePerWindow.java | 2 +-
.../beam/examples/complete/AutoComplete.java | 4 +-
.../org/apache/beam/examples/complete/TfIdf.java | 2 +-
.../examples/complete/TopWikipediaSessions.java | 2 +-
.../beam/examples/complete/TrafficRoutes.java | 4 +-
.../beam/examples/complete/game/LeaderBoard.java | 2 +-
.../examples/complete/game/StatefulTeamScore.java | 4 +-
.../examples/complete/game/injector/Injector.java | 2 +-
.../complete/game/injector/InjectorUtils.java | 2 +-
.../game/injector/RetryHttpInitializerWrapper.java | 2 +-
.../examples/complete/game/utils/WriteToText.java | 2 +-
.../apache/beam/examples/snippets/Snippets.java | 2 +-
.../kernel/SubProcessCommandLineArgs.java | 2 +-
.../subprocess/utils/CallingSubProcessUtils.java | 2 +-
.../beam/examples/DebuggingWordCountTest.java | 2 +-
.../apache/beam/examples/MinimalWordCountTest.java | 2 +-
.../apache/beam/examples/WindowedWordCountIT.java | 6 +-
.../examples/complete/TrafficMaxLaneFlowIT.java | 2 +-
.../beam/examples/complete/TrafficRoutesIT.java | 2 +-
.../examples/complete/game/LeaderBoardTest.java | 2 +-
.../beam/examples/complete/game/UserScoreTest.java | 2 +-
.../examples/cookbook/BigQueryTornadoesTest.java | 2 +-
.../examples/cookbook/MaxPerKeyExamplesTest.java | 2 +-
.../beam/examples/cookbook/TriggerExampleTest.java | 4 +-
.../subprocess/ExampleEchoPipelineTest.java | 2 +-
gradle/wrapper/gradle-wrapper.jar | Bin 56177 -> 56177 bytes
gradle/wrapper/gradle-wrapper.properties | 2 +-
.../src/main/groovy/MobileGamingCommands.groovy | 20 +-
.../main/groovy/mobilegaming-java-dataflow.groovy | 100 ++--
.../org/apache/beam/runners/apex/ApexRunner.java | 6 +-
.../beam/runners/apex/ApexRunnerRegistrar.java | 2 +-
.../apache/beam/runners/apex/ApexYarnLauncher.java | 8 +-
.../translation/FlattenPCollectionTranslator.java | 4 +-
.../runners/apex/translation/ParDoTranslator.java | 4 +-
.../apex/translation/TranslationContext.java | 4 +-
.../operators/ApexGroupByKeyOperator.java | 4 +-
.../translation/operators/ApexParDoOperator.java | 8 +-
.../operators/ApexProcessFnOperator.java | 4 +-
.../operators/ApexReadUnboundedInputOperator.java | 2 +-
.../translation/operators/ApexTimerInternals.java | 8 +-
.../apex/translation/utils/ApexStateInternals.java | 4 +-
.../apex/translation/utils/ApexStreamTuple.java | 2 +-
.../translation/utils/CoderAdapterStreamCodec.java | 2 +-
.../apache/beam/runners/apex/ApexRunnerTest.java | 2 +-
.../beam/runners/apex/examples/WordCountTest.java | 2 +-
.../translation/ApexGroupByKeyOperatorTest.java | 2 +-
.../FlattenPCollectionTranslatorTest.java | 4 +-
.../apex/translation/GroupByKeyTranslatorTest.java | 4 +-
.../apex/translation/ParDoTranslatorTest.java | 4 +-
.../translation/ReadUnboundTranslatorTest.java | 10 +-
.../apex/translation/SideInputTranslationTest.java | 2 +-
.../operators/ApexTimerInternalsTest.java | 2 +-
runners/core-construction-java/build.gradle | 4 +-
.../core/construction/ArtifactServiceStager.java | 8 +-
.../core/construction/CoderTranslation.java | 10 +-
.../core/construction/CoderTranslators.java | 2 +-
.../core/construction/CombineTranslation.java | 6 +-
.../CreatePCollectionViewTranslation.java | 2 +-
.../construction/DeduplicatedFlattenFactory.java | 2 +-
.../construction/EmptyFlattenAsCreateFactory.java | 2 +-
.../runners/core/construction/Environments.java | 6 +-
.../construction/ExecutableStageTranslation.java | 10 +-
.../core/construction/JavaReadViaImpulse.java | 2 +-
.../core/construction/ModelCoderRegistrar.java | 12 +-
.../runners/core/construction/ModelCoders.java | 4 +-
.../construction/PCollectionViewTranslation.java | 2 +-
.../core/construction/PTransformMatchers.java | 2 +-
.../core/construction/PTransformReplacements.java | 4 +-
.../core/construction/PTransformTranslation.java | 12 +-
.../core/construction/ParDoTranslation.java | 10 +-
...java => PipelineOptionsSerializationUtils.java} | 34 +-
.../construction/PipelineOptionsTranslation.java | 4 +-
.../core/construction/PipelineResources.java | 10 +-
.../core/construction/PipelineTranslation.java | 6 +-
.../runners/core/construction/PrimitiveCreate.java | 2 +-
.../runners/core/construction/ReadTranslation.java | 4 +-
.../core/construction/RehydratedComponents.java | 8 +-
.../core/construction/ReplacementOutputs.java | 6 +-
.../runners/core/construction/SdkComponents.java | 12 +-
.../construction/SerializablePipelineOptions.java | 26 +-
.../runners/core/construction/SplittableParDo.java | 6 +-
.../construction/SplittableParDoNaiveBounded.java | 4 +-
.../core/construction/TestStreamTranslation.java | 2 +-
.../runners/core/construction/TransformInputs.java | 4 +-
.../core/construction/TriggerTranslation.java | 4 +-
.../UnboundedReadFromBoundedSource.java | 10 +-
.../core/construction/WindowIntoTranslation.java | 2 +-
.../core/construction/WriteFilesTranslation.java | 10 +-
.../core/construction/graph/FusedPipeline.java | 2 +-
.../graph/GreedyPCollectionFusers.java | 6 +-
.../construction/graph/GreedyPipelineFuser.java | 14 +-
.../core/construction/graph/GreedyStageFuser.java | 4 +-
.../graph/ImmutableExecutableStage.java | 2 +-
.../runners/core/construction/graph/Networks.java | 24 +-
.../construction/graph/OutputDeduplicator.java | 6 +-
.../core/construction/graph/PipelineValidator.java | 6 +-
.../core/construction/graph/ProtoOverrides.java | 2 +-
.../core/construction/graph/QueryablePipeline.java | 16 +-
.../core/construction/metrics/MetricFiltering.java | 2 +-
.../construction/ArtifactServiceStagerTest.java | 6 +-
.../core/construction/CoderTranslationTest.java | 4 +-
.../core/construction/CombineTranslationTest.java | 4 +-
.../runners/core/construction/CommonCoderTest.java | 14 +-
.../CreatePCollectionViewTranslationTest.java | 2 +-
.../construction/ForwardingPTransformTest.java | 2 +-
.../InMemoryArtifactStagerService.java | 6 +-
.../core/construction/JavaReadViaImpulseTest.java | 2 +-
.../construction/PCollectionTranslationTest.java | 2 +-
.../core/construction/PTransformMatchersTest.java | 4 +-
.../construction/PTransformReplacementsTest.java | 2 +-
.../construction/PTransformTranslationTest.java | 2 +-
.../core/construction/ParDoTranslationTest.java | 2 +-
.../PipelineOptionsTranslationTest.java | 2 +-
.../core/construction/PipelineResourcesTest.java | 2 +-
.../core/construction/PipelineTranslationTest.java | 4 +-
.../core/construction/ReadTranslationTest.java | 2 +-
.../core/construction/ReplacementOutputsTest.java | 4 +-
.../construction/TestStreamTranslationTest.java | 2 +-
.../core/construction/TriggerTranslationTest.java | 2 +-
.../UnboundedReadFromBoundedSourceTest.java | 6 +-
.../construction/WindowIntoTranslationTest.java | 4 +-
.../WindowingStrategyTranslationTest.java | 2 +-
.../construction/WriteFilesTranslationTest.java | 2 +-
.../construction/graph/ExecutableStageMatcher.java | 2 +-
.../construction/graph/ExecutableStageTest.java | 2 +-
.../core/construction/graph/FusedPipelineTest.java | 4 +-
.../graph/GreedyPipelineFuserTest.java | 2 +-
.../construction/graph/GreedyStageFuserTest.java | 4 +-
.../core/construction/graph/NetworksTest.java | 20 +-
.../construction/graph/OutputDeduplicatorTest.java | 4 +-
.../construction/graph/ProtoOverridesTest.java | 2 +-
.../construction/graph/QueryablePipelineTest.java | 4 +-
runners/core-java/build.gradle | 2 +-
.../apache/beam/runners/core/ActiveWindowSet.java | 2 +-
.../beam/runners/core/GlobalCombineFnRunners.java | 2 +-
.../runners/core/GroupByKeyViaGroupByKeyOnly.java | 2 +-
.../core/InMemoryMultimapSideInputView.java | 6 +-
.../beam/runners/core/InMemoryStateInternals.java | 6 +-
.../beam/runners/core/InMemoryTimerInternals.java | 12 +-
.../beam/runners/core/KeyedWorkItemCoder.java | 2 +-
.../apache/beam/runners/core/KeyedWorkItems.java | 4 +-
.../runners/core/LateDataDroppingDoFnRunner.java | 4 +-
.../apache/beam/runners/core/LateDataUtils.java | 4 +-
.../beam/runners/core/MergingActiveWindowSet.java | 10 +-
.../runners/core/NonMergingActiveWindowSet.java | 4 +-
.../beam/runners/core/NullSideInputReader.java | 2 +-
...TimeBoundedSplittableProcessElementInvoker.java | 8 +-
.../apache/beam/runners/core/PaneInfoTracker.java | 4 +-
.../beam/runners/core/PeekingReiterator.java | 4 +-
.../apache/beam/runners/core/ProcessFnRunner.java | 4 +-
.../beam/runners/core/ReduceFnContextFactory.java | 6 +-
.../apache/beam/runners/core/ReduceFnRunner.java | 10 +-
.../apache/beam/runners/core/SideInputHandler.java | 2 +-
.../apache/beam/runners/core/SimpleDoFnRunner.java | 12 +-
.../core/SimplePushbackSideInputDoFnRunner.java | 4 +-
.../core/SplittableParDoViaKeyedWorkItems.java | 4 +-
.../core/SplittableProcessElementInvoker.java | 2 +-
.../apache/beam/runners/core/StateNamespaces.java | 2 +-
.../org/apache/beam/runners/core/StateTable.java | 6 +-
.../org/apache/beam/runners/core/StateTags.java | 4 +-
.../apache/beam/runners/core/SystemReduceFn.java | 2 +-
.../apache/beam/runners/core/TimerInternals.java | 2 +-
.../apache/beam/runners/core/WatermarkHold.java | 4 +-
.../beam/runners/core/metrics/MetricUpdates.java | 2 +-
.../runners/core/metrics/MetricsContainerImpl.java | 4 +-
.../core/metrics/MetricsContainerStepMap.java | 6 +-
.../beam/runners/core/metrics/MetricsMap.java | 4 +-
.../beam/runners/core/metrics/MetricsPusher.java | 10 +-
.../runners/core/metrics/MetricsTranslation.java | 6 +-
.../beam/runners/core/metrics/NoOpMetricsSink.java | 4 +-
.../core/metrics/SimpleMonitoringInfoBuilder.java | 2 +-
.../core/serialization}/Base64Serializer.java | 4 +-
.../runners/core/serialization/package-info.java | 16 +-
.../core/triggers/AfterAllStateMachine.java | 6 +-
.../AfterDelayFromFirstElementStateMachine.java | 2 +-
.../core/triggers/AfterEachStateMachine.java | 6 +-
.../core/triggers/AfterFirstStateMachine.java | 6 +-
...fterSynchronizedProcessingTimeStateMachine.java | 2 +-
.../core/triggers/AfterWatermarkStateMachine.java | 4 +-
.../triggers/ExecutableTriggerStateMachine.java | 4 +-
.../runners/core/triggers/FinishedTriggersSet.java | 2 +-
.../core/triggers/OrFinallyStateMachine.java | 2 +-
.../runners/core/triggers/TriggerStateMachine.java | 2 +-
.../TriggerStateMachineContextFactory.java | 6 +-
.../core/triggers/TriggerStateMachineRunner.java | 6 +-
.../core/InMemoryMultimapSideInputViewTest.java | 2 +-
.../beam/runners/core/KeyedWorkItemCoderTest.java | 2 +-
.../core/LateDataDroppingDoFnRunnerTest.java | 4 +-
.../runners/core/MergingActiveWindowSetTest.java | 4 +-
...BoundedSplittableProcessElementInvokerTest.java | 2 +-
.../beam/runners/core/ReduceFnRunnerTest.java | 2 +-
.../apache/beam/runners/core/ReduceFnTester.java | 10 +-
.../beam/runners/core/SideInputHandlerTest.java | 2 +-
.../beam/runners/core/SimpleDoFnRunnerTest.java | 4 +-
.../SimplePushbackSideInputDoFnRunnerTest.java | 2 +-
.../runners/core/SplittableParDoProcessFnTest.java | 2 +-
.../beam/runners/core/StateInternalsTest.java | 2 +-
.../beam/runners/core/StatefulDoFnRunnerTest.java | 4 +-
.../apache/beam/runners/core/WindowMatchers.java | 2 +-
.../beam/runners/core/WindowMatchersTest.java | 2 +-
.../runners/core/metrics/MetricsPusherTest.java | 7 +-
.../core/metrics/MetricsTranslationTest.java | 4 +-
.../beam/runners/core/metrics/TestMetricsSink.java | 4 +-
.../core/triggers/StubTriggerStateMachine.java | 2 +-
.../core/triggers/TriggerStateMachineTester.java | 8 +-
.../core/triggers/TriggerStateMachinesTest.java | 2 +-
runners/direct-java/build.gradle | 6 +-
.../direct/BoundedReadEvaluatorFactory.java | 12 +-
.../beam/runners/direct/CommittedResult.java | 2 +-
.../direct/CopyOnAccessInMemoryStateInternals.java | 6 +-
.../apache/beam/runners/direct/DirectGraph.java | 2 +-
.../beam/runners/direct/DirectGraphVisitor.java | 8 +-
.../beam/runners/direct/DirectGroupByKey.java | 2 +-
.../direct/DirectGroupByKeyOverrideFactory.java | 2 +-
.../apache/beam/runners/direct/DirectMetrics.java | 2 +-
.../beam/runners/direct/DirectRegistrar.java | 2 +-
.../apache/beam/runners/direct/DirectRunner.java | 14 +-
.../runners/direct/DirectTransformExecutor.java | 4 +-
.../beam/runners/direct/DoFnLifecycleManager.java | 10 +-
.../beam/runners/direct/EvaluationContext.java | 12 +-
.../direct/ExecutorServiceParallelExecutor.java | 16 +-
.../runners/direct/FlattenEvaluatorFactory.java | 2 +-
.../direct/GroupAlsoByWindowEvaluatorFactory.java | 4 +-
.../direct/GroupByKeyOnlyEvaluatorFactory.java | 4 +-
.../direct/ImmutabilityCheckingBundleFactory.java | 6 +-
.../runners/direct/ImmutableListBundleFactory.java | 6 +-
.../runners/direct/ImpulseEvaluatorFactory.java | 4 +-
.../runners/direct/KeyedPValueTrackingVisitor.java | 4 +-
.../beam/runners/direct/MultiStepCombine.java | 8 +-
.../apache/beam/runners/direct/ParDoEvaluator.java | 4 +-
.../beam/runners/direct/ParDoEvaluatorFactory.java | 6 +-
.../runners/direct/ParDoMultiOverrideFactory.java | 4 +-
.../beam/runners/direct/QuiescenceDriver.java | 4 +-
.../beam/runners/direct/RootProviderRegistry.java | 4 +-
.../beam/runners/direct/SideInputContainer.java | 18 +-
.../SplittableProcessElementsEvaluatorFactory.java | 8 +-
.../direct/StatefulParDoEvaluatorFactory.java | 10 +-
.../org/apache/beam/runners/direct/StepAndKey.java | 2 +-
.../beam/runners/direct/StepTransformResult.java | 2 +-
.../runners/direct/TestStreamEvaluatorFactory.java | 6 +-
.../runners/direct/TransformEvaluatorRegistry.java | 6 +-
.../runners/direct/TransformExecutorServices.java | 2 +-
.../runners/direct/UnboundedReadDeduplicator.java | 6 +-
.../direct/UnboundedReadEvaluatorFactory.java | 6 +-
.../beam/runners/direct/ViewEvaluatorFactory.java | 2 +-
.../runners/direct/WatermarkCallbackExecutor.java | 4 +-
.../beam/runners/direct/WatermarkManager.java | 24 +-
.../runners/direct/WindowEvaluatorFactory.java | 2 +-
.../runners/direct/WriteWithShardingFactory.java | 6 +-
.../runners/direct/portable/CommittedResult.java | 2 +-
.../CopyOnAccessInMemoryStateInternals.java | 6 +-
.../runners/direct/portable/DirectGroupByKey.java | 2 +-
.../runners/direct/portable/DirectMetrics.java | 6 +-
.../direct/portable/DirectTransformExecutor.java | 2 +-
.../runners/direct/portable/EvaluationContext.java | 12 +-
.../portable/ExecutorServiceParallelExecutor.java | 16 +-
.../direct/portable/FlattenEvaluatorFactory.java | 2 +-
.../GroupAlsoByWindowEvaluatorFactory.java | 4 +-
.../portable/GroupByKeyOnlyEvaluatorFactory.java | 4 +-
.../portable/ImmutableListBundleFactory.java | 8 +-
.../direct/portable/ImpulseEvaluatorFactory.java | 4 +-
.../runners/direct/portable/QuiescenceDriver.java | 4 +-
.../runners/direct/portable/ReferenceRunner.java | 16 +-
.../portable/RemoteStageEvaluatorFactory.java | 2 +-
.../direct/portable/RootProviderRegistry.java | 4 +-
.../SplittableRemoteStageEvaluatorFactory.java | 2 +-
.../beam/runners/direct/portable/StepAndKey.java | 2 +-
.../direct/portable/StepTransformResult.java | 2 +-
.../portable/TransformEvaluatorRegistry.java | 6 +-
.../direct/portable/TransformExecutorServices.java | 2 +-
.../direct/portable/WatermarkCallbackExecutor.java | 4 +-
.../direct/portable/WindowEvaluatorFactory.java | 2 +-
.../artifact/LocalArtifactStagingLocation.java | 4 +-
.../LocalFileSystemArtifactStagerService.java | 6 +-
.../portable/job/ReferenceRunnerJobService.java | 6 +-
.../direct/BoundedReadEvaluatorFactoryTest.java | 4 +-
.../runners/direct/CloningBundleFactoryTest.java | 4 +-
.../beam/runners/direct/CommittedResultTest.java | 4 +-
.../CopyOnAccessInMemoryStateInternalsTest.java | 2 +-
.../runners/direct/DirectGraphVisitorTest.java | 2 +-
.../beam/runners/direct/DirectMetricsTest.java | 2 +-
.../beam/runners/direct/DirectRegistrarTest.java | 4 +-
.../runners/direct/DirectRunnerApiSurfaceTest.java | 2 +-
.../beam/runners/direct/DirectRunnerTest.java | 4 +-
.../direct/DirectTransformExecutorTest.java | 6 +-
.../runners/direct/DoFnLifecycleManagerTest.java | 2 +-
.../runners/direct/DoFnLifecycleManagersTest.java | 2 +-
.../beam/runners/direct/EvaluationContextTest.java | 4 +-
.../ExecutorServiceParallelExecutorTest.java | 4 +-
.../direct/FlattenEvaluatorFactoryTest.java | 2 +-
.../direct/GroupByKeyEvaluatorFactoryTest.java | 6 +-
.../direct/GroupByKeyOnlyEvaluatorFactoryTest.java | 6 +-
.../direct/ImmutableListBundleFactoryTest.java | 2 +-
.../direct/ImpulseEvaluatorFactoryTest.java | 2 +-
.../org/apache/beam/runners/direct/MockClock.java | 2 +-
.../beam/runners/direct/ParDoEvaluatorTest.java | 6 +-
.../runners/direct/SideInputContainerTest.java | 6 +-
.../direct/StatefulParDoEvaluatorFactoryTest.java | 4 +-
.../direct/TestStreamEvaluatorFactoryTest.java | 2 +-
.../direct/TransformExecutorServicesTest.java | 2 +-
.../direct/UnboundedReadDeduplicatorTest.java | 8 +-
.../direct/UnboundedReadEvaluatorFactoryTest.java | 14 +-
.../runners/direct/ViewEvaluatorFactoryTest.java | 2 +-
.../beam/runners/direct/WatermarkManagerTest.java | 6 +-
.../runners/direct/WindowEvaluatorFactoryTest.java | 6 +-
.../direct/WriteWithShardingFactoryTest.java | 2 +-
.../BundleFactoryOutputReceiverFactoryTest.java | 4 +-
.../direct/portable/CommittedResultTest.java | 4 +-
.../CopyOnAccessInMemoryStateInternalsTest.java | 2 +-
.../runners/direct/portable/DirectMetricsTest.java | 2 +-
.../portable/DirectTransformExecutorTest.java | 6 +-
.../direct/portable/EvaluationContextTest.java | 6 +-
.../portable/FlattenEvaluatorFactoryTest.java | 2 +-
.../GroupByKeyOnlyEvaluatorFactoryTest.java | 6 +-
.../portable/ImmutableListBundleFactoryTest.java | 2 +-
.../portable/ImpulseEvaluatorFactoryTest.java | 2 +-
.../beam/runners/direct/portable/MockClock.java | 2 +-
.../runners/direct/portable/PortableGraphTest.java | 2 +-
.../direct/portable/ReferenceRunnerTest.java | 4 +-
.../portable/RemoteStageEvaluatorFactoryTest.java | 6 +-
.../portable/TransformExecutorServicesTest.java | 2 +-
.../portable/WindowEvaluatorFactoryTest.java | 6 +-
.../artifact/LocalArtifactStagingLocationTest.java | 2 +-
.../LocalFileSystemArtifactStagerServiceTest.java | 2 +-
.../job/ReferenceRunnerJobServiceTest.java | 2 +-
runners/extensions-java/metrics/build.gradle | 2 +-
.../extensions/metrics/MetricsGraphiteSink.java | 4 +-
.../extensions/metrics/MetricsHttpSink.java | 4 +-
.../metrics/MetricsGraphiteSinkTest.java | 6 +-
.../extensions/metrics/MetricsHttpSinkTest.java | 6 +-
runners/flink/flink_runner.gradle | 13 +-
runners/flink/job-server/flink_job_server.gradle | 14 +-
.../runners/flink/CreateStreamingFlinkView.java | 2 +-
.../FlinkBatchPortablePipelineTranslator.java | 12 +-
.../flink/FlinkBatchTransformTranslators.java | 8 +-
.../flink/FlinkBatchTranslationContext.java | 2 +-
.../runners/flink/FlinkExecutionEnvironments.java | 7 +-
.../beam/runners/flink/FlinkJobInvocation.java | 18 +-
.../apache/beam/runners/flink/FlinkJobInvoker.java | 2 +-
.../beam/runners/flink/FlinkJobServerDriver.java | 8 +-
.../flink/FlinkPipelineExecutionEnvironment.java | 2 +-
.../beam/runners/flink/FlinkPipelineOptions.java | 9 +
.../org/apache/beam/runners/flink/FlinkRunner.java | 7 +-
.../beam/runners/flink/FlinkRunnerRegistrar.java | 2 +-
.../flink/FlinkStreamingPipelineTranslator.java | 2 +-
.../FlinkStreamingPortablePipelineTranslator.java | 12 +-
.../flink/FlinkStreamingTransformTranslators.java | 6 +-
.../flink/FlinkStreamingTranslationContext.java | 4 +-
.../runners/flink/FlinkTransformOverrides.java | 4 +-
.../flink/metrics/FlinkMetricContainer.java | 2 +-
.../functions/AbstractFlinkCombineRunner.java | 2 +-
.../translation/functions/FlinkAssignContext.java | 2 +-
.../FlinkBatchSideInputHandlerFactory.java | 8 +-
.../FlinkDefaultExecutableStageContext.java | 9 +-
.../translation/functions/FlinkDoFnFunction.java | 2 +-
.../functions/FlinkExecutableStageFunction.java | 6 +-
.../functions/FlinkSideInputReader.java | 4 +-
.../functions/FlinkStatefulDoFnFunction.java | 2 +-
.../FlinkStreamingSideInputHandlerFactory.java | 6 +-
.../functions/HashingFlinkCombineRunner.java | 2 +-
...CountingFlinkExecutableStageContextFactory.java | 4 +-
.../functions/SideInputInitializer.java | 2 +-
.../functions/SortingFlinkCombineRunner.java | 4 +-
.../translation/types/CoderTypeInformation.java | 2 +-
.../utils/FlinkPipelineTranslatorUtils.java | 6 +-
.../translation/wrappers/ImpulseInputFormat.java | 2 +-
.../wrappers/streaming/DoFnOperator.java | 6 +-
.../streaming/ExecutableStageDoFnOperator.java | 8 +-
.../NonKeyedPushedBackElementsHandler.java | 2 +-
.../streaming/SingletonKeyedWorkItemCoder.java | 2 +-
.../wrappers/streaming/SplittableDoFnOperator.java | 2 +-
.../streaming/io/UnboundedSocketSource.java | 2 +-
.../streaming/io/UnboundedSourceWrapper.java | 2 +-
.../streaming/state/FlinkSplitStateInternals.java | 2 +-
.../streaming/state/FlinkStateInternals.java | 8 +-
.../runners/flink/FlinkJobServerDriverTest.java | 2 +-
.../FlinkPipelineExecutionEnvironmentTest.java | 2 +-
.../beam/runners/flink/PipelineOptionsTest.java | 1 +
.../beam/runners/flink/PortableExecutionTest.java | 6 +-
.../runners/flink/PortableStateExecutionTest.java | 4 +-
.../runners/flink/PortableTimersExecutionTest.java | 4 +-
.../runners/flink/ReadSourceStreamingTest.java | 2 +-
.../apache/beam/runners/flink/ReadSourceTest.java | 2 +-
.../runners/flink/streaming/DoFnOperatorTest.java | 8 +-
.../streaming/ExecutableStageDoFnOperatorTest.java | 4 +-
.../flink/streaming/GroupByNullKeyTest.java | 2 +-
.../flink/streaming/StreamRecordStripper.java | 4 +-
.../flink/streaming/TopWikipediaSessionsTest.java | 2 +-
.../flink/streaming/WindowDoFnOperatorTest.java | 2 +-
.../FlinkPipelineTranslatorUtilsTest.java | 4 +-
.../FlinkExecutableStageFunctionTest.java | 2 +-
...tingFlinkExecutableStageContextFactoryTest.java | 2 +-
.../streaming/io/UnboundedSourceWrapperTest.java | 2 +-
runners/gearpump/build.gradle | 10 +-
.../runners/gearpump/GearpumpPipelineOptions.java | 12 +-
.../runners/gearpump/GearpumpPipelineResult.java | 18 +-
.../beam/runners/gearpump/GearpumpRunner.java | 33 +-
.../runners/gearpump/GearpumpRunnerRegistrar.java | 2 +-
.../beam/runners/gearpump/TestGearpumpRunner.java | 19 +-
.../CreateGearpumpPCollectionViewTranslator.java | 2 +-
.../translators/CreateStreamingGearpumpView.java | 2 +-
.../translators/FlattenPCollectionsTranslator.java | 6 +-
.../translators/GearpumpPipelineTranslator.java | 4 +-
.../gearpump/translators/GroupByKeyTranslator.java | 22 +-
.../translators/ParDoMultiOutputTranslator.java | 4 +-
.../translators/ReadBoundedTranslator.java | 4 +-
.../translators/ReadUnboundedTranslator.java | 4 +-
.../gearpump/translators/TranslationContext.java | 12 +-
.../translators/WindowAssignTranslator.java | 6 +-
.../translators/functions/DoFnFunction.java | 13 +-
.../gearpump/translators/io/GearpumpSource.java | 10 +-
.../translators/utils/TranslatorUtils.java | 17 +-
.../gearpump/GearpumpRunnerRegistrarTest.java | 2 +-
.../beam/runners/gearpump/PipelineOptionsTest.java | 9 +-
...reateGearpumpPCollectionViewTranslatorTest.java | 2 +-
.../FlattenPCollectionsTranslatorTest.java | 6 +-
.../translators/GroupByKeyTranslatorTest.java | 8 +-
.../translators/ReadBoundedTranslatorTest.java | 4 +-
.../translators/ReadUnboundedTranslatorTest.java | 4 +-
.../translators/WindowAssignTranslatorTest.java | 2 +-
.../translators/io/GearpumpSourceTest.java | 8 +-
.../gearpump/translators/io/ValueSoureTest.java | 14 +-
.../translators/utils/TranslatorUtilsTest.java | 4 +-
runners/google-cloud-dataflow-java/build.gradle | 12 +-
.../dataflow/BatchStatefulParDoOverrides.java | 2 +-
.../beam/runners/dataflow/BatchViewOverrides.java | 22 +-
.../beam/runners/dataflow/DataflowClient.java | 2 +-
.../beam/runners/dataflow/DataflowMetrics.java | 6 +-
.../dataflow/DataflowPTransformMatchers.java | 2 +-
.../beam/runners/dataflow/DataflowPipelineJob.java | 10 +-
.../dataflow/DataflowPipelineRegistrar.java | 2 +-
.../dataflow/DataflowPipelineTranslator.java | 14 +-
.../beam/runners/dataflow/DataflowRunner.java | 20 +-
.../beam/runners/dataflow/DataflowRunnerInfo.java | 4 +-
.../dataflow/PrimitiveParDoSingleFactory.java | 4 +-
.../beam/runners/dataflow/TestDataflowRunner.java | 8 +-
.../runners/dataflow/internal/CustomSources.java | 4 +-
.../beam/runners/dataflow/internal/IsmFormat.java | 12 +-
.../options/DataflowWorkerLoggingOptions.java | 2 +-
.../beam/runners/dataflow/util/CloudKnownType.java | 3 +-
.../beam/runners/dataflow/util/CloudObject.java | 2 +-
.../runners/dataflow/util/CloudObjectKinds.java | 1 -
.../dataflow/util/CloudObjectTranslators.java | 10 +-
.../beam/runners/dataflow/util/CloudObjects.java | 6 +-
.../runners/dataflow/util/DataflowTemplateJob.java | 2 +-
.../runners/dataflow/util/DataflowTransport.java | 2 +-
...DefaultCoderCloudObjectTranslatorRegistrar.java | 10 +-
.../beam/runners/dataflow/util/GcsStager.java | 6 +-
.../beam/runners/dataflow/util/MonitoringUtil.java | 6 +-
.../runners/dataflow/util/OutputReference.java | 2 +-
.../beam/runners/dataflow/util/PackageUtil.java | 18 +-
.../runners/dataflow/util/RandomAccessData.java | 10 +-
.../SerializableCoderCloudObjectTranslator.java | 2 +-
.../beam/runners/dataflow/util/TimeUtil.java | 2 +-
.../dataflow/BatchStatefulParDoOverridesTest.java | 2 +-
.../runners/dataflow/BatchViewOverridesTest.java | 4 +-
.../beam/runners/dataflow/DataflowMetricsTest.java | 6 +-
.../runners/dataflow/DataflowPipelineJobTest.java | 6 +-
.../dataflow/DataflowPipelineRegistrarTest.java | 4 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 8 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 4 +-
.../dataflow/PrimitiveParDoSingleFactoryTest.java | 2 +-
.../runners/dataflow/TestDataflowRunnerTest.java | 6 +-
.../options/DataflowPipelineOptionsTest.java | 2 +-
.../options/DataflowWorkerLoggingOptionsTest.java | 2 +-
.../runners/dataflow/util/CloudObjectsTest.java | 4 +-
.../runners/dataflow/util/PackageUtilTest.java | 12 +-
.../dataflow/util/RandomAccessDataTest.java | 2 +-
.../google-cloud-dataflow-java/worker/build.gradle | 4 +-
.../worker/legacy-worker/build.gradle | 4 +-
.../worker/ApplianceShuffleEntryReader.java | 2 +-
.../worker/AssignWindowsParDoFnFactory.java | 4 +-
.../runners/dataflow/worker/AvroByteReader.java | 2 +-
.../dataflow/worker/AvroByteReaderFactory.java | 2 +-
.../beam/runners/dataflow/worker/AvroByteSink.java | 2 +-
.../dataflow/worker/AvroByteSinkFactory.java | 2 +-
.../dataflow/worker/BatchDataflowWorker.java | 10 +-
.../dataflow/worker/BatchModeExecutionContext.java | 10 +-
.../worker/BeamFnMapTaskExecutorFactory.java | 14 +-
.../runners/dataflow/worker/ByteStringCoder.java | 2 +-
.../worker/ChunkingShuffleBatchReader.java | 2 +-
.../dataflow/worker/CombineValuesFnFactory.java | 2 +-
.../beam/runners/dataflow/worker/ConcatReader.java | 6 +-
.../worker/ContextActivationObserverRegistry.java | 6 +-
.../dataflow/worker/CounterShortIdCache.java | 6 +-
.../CreateIsmShardKeyAndSortKeyDoFnFactory.java | 4 +-
.../runners/dataflow/worker/DataflowApiUtils.java | 4 +-
.../worker/DataflowBatchWorkerHarness.java | 4 +-
.../worker/DataflowElementExecutionTracker.java | 10 +-
.../dataflow/worker/DataflowExecutionContext.java | 6 +-
.../worker/DataflowMapTaskExecutorFactory.java | 2 +-
.../dataflow/worker/DataflowOperationContext.java | 6 +-
.../dataflow/worker/DataflowOutputCounter.java | 2 +-
.../dataflow/worker/DataflowProcessFnRunner.java | 4 +-
.../dataflow/worker/DataflowRunnerHarness.java | 2 +-
.../worker/DataflowSideInputReadCounter.java | 2 +-
.../dataflow/worker/DataflowSystemMetrics.java | 2 +-
.../worker/DataflowWorkProgressUpdater.java | 2 +-
.../dataflow/worker/DataflowWorkUnitClient.java | 15 +-
.../dataflow/worker/DefaultParDoFnFactory.java | 2 +-
.../runners/dataflow/worker/ExecutionStateKey.java | 4 +-
.../dataflow/worker/ExecutionStateRegistry.java | 4 +-
.../runners/dataflow/worker/ExperimentContext.java | 4 +-
...FetchAndFilterStreamingSideInputsOperation.java | 2 +-
.../beam/runners/dataflow/worker/Filepatterns.java | 2 +-
.../dataflow/worker/FnApiWindowMappingFn.java | 8 +-
.../worker/GroupAlsoByWindowParDoFnFactory.java | 4 +-
.../dataflow/worker/GroupAlsoByWindowsParDoFn.java | 2 +-
.../dataflow/worker/GroupingShuffleReader.java | 4 +-
.../worker/GroupingShuffleReaderFactory.java | 2 +-
...ingShuffleReaderWithFaultyBytesReadCounter.java | 2 +-
.../runners/dataflow/worker/InMemoryReader.java | 8 +-
.../dataflow/worker/InMemoryReaderFactory.java | 2 +-
.../worker/IntrinsicMapTaskExecutorFactory.java | 10 +-
.../beam/runners/dataflow/worker/IsmReader.java | 2 +-
.../runners/dataflow/worker/IsmReaderFactory.java | 6 +-
.../runners/dataflow/worker/IsmReaderImpl.java | 32 +-
.../dataflow/worker/IsmSideInputReader.java | 26 +-
.../beam/runners/dataflow/worker/IsmSink.java | 12 +-
.../runners/dataflow/worker/IsmSinkFactory.java | 6 +-
.../worker/LazilyInitializedSideInputReader.java | 4 +-
.../worker/MetricTrackingWindmillServerStub.java | 2 +-
.../dataflow/worker/MetricsContainerRegistry.java | 2 +-
...nmentContextActivationObserverRegistration.java | 2 +-
.../beam/runners/dataflow/worker/OrderedCode.java | 4 +-
.../worker/PairWithConstantKeyDoFnFactory.java | 2 +-
.../dataflow/worker/PartialGroupByKeyParDoFns.java | 6 +-
.../dataflow/worker/PartitioningShuffleReader.java | 2 +-
.../worker/PartitioningShuffleReaderFactory.java | 4 +-
.../beam/runners/dataflow/worker/PubsubReader.java | 2 +-
.../beam/runners/dataflow/worker/PubsubSink.java | 2 +-
.../beam/runners/dataflow/worker/ReaderCache.java | 10 +-
.../runners/dataflow/worker/ReaderRegistry.java | 8 +-
.../ReifyTimestampAndWindowsParDoFnFactory.java | 2 +-
...HarnessCoderCloudObjectTranslatorRegistrar.java | 4 +-
.../dataflow/worker/SdkHarnessRegistries.java | 2 +-
.../beam/runners/dataflow/worker/ShuffleSink.java | 4 +-
.../dataflow/worker/ShuffleSinkFactory.java | 2 +-
.../runners/dataflow/worker/SimpleParDoFn.java | 8 +-
.../beam/runners/dataflow/worker/SinkRegistry.java | 6 +-
.../dataflow/worker/SizeReportingSinkWrapper.java | 2 +-
.../worker/SourceOperationExecutorFactory.java | 2 +-
.../beam/runners/dataflow/worker/StateFetcher.java | 12 +-
.../dataflow/worker/StreamingDataflowWorker.java | 28 +-
.../worker/StreamingGroupAlsoByWindowsDoFns.java | 2 +-
.../StreamingKeyedWorkItemSideInputDoFnRunner.java | 6 +-
.../worker/StreamingModeExecutionContext.java | 14 +-
.../StreamingPCollectionViewWriterDoFnFactory.java | 4 +-
.../StreamingPCollectionViewWriterParDoFn.java | 4 +-
.../dataflow/worker/StreamingSideInputFetcher.java | 6 +-
.../worker/StreamingStepMetricsContainer.java | 6 +-
.../worker/ToIsmRecordForMultimapDoFnFactory.java | 4 +-
.../dataflow/worker/UngroupedShuffleReader.java | 2 +-
.../worker/UngroupedShuffleReaderFactory.java | 2 +-
.../dataflow/worker/UngroupedWindmillReader.java | 2 +-
.../dataflow/worker/UserParDoFnFactory.java | 6 +-
.../runners/dataflow/worker/ValuesDoFnFactory.java | 2 +-
.../beam/runners/dataflow/worker/Weighers.java | 2 +-
.../dataflow/worker/WindmillKeyedWorkItem.java | 8 +-
.../worker/WindmillReaderIteratorBase.java | 2 +-
.../beam/runners/dataflow/worker/WindmillSink.java | 2 +-
.../dataflow/worker/WindmillStateCache.java | 10 +-
.../dataflow/worker/WindmillStateInternals.java | 12 +-
.../dataflow/worker/WindmillStateReader.java | 18 +-
.../runners/dataflow/worker/WindmillTimeUtils.java | 2 +-
.../dataflow/worker/WindmillTimerInternals.java | 12 +-
.../dataflow/worker/WindowingWindmillReader.java | 2 +-
.../dataflow/worker/WorkItemStatusClient.java | 10 +-
.../runners/dataflow/worker/WorkUnitClient.java | 2 +-
.../dataflow/worker/WorkerCustomSources.java | 10 +-
.../worker/WorkerUncaughtExceptionHandler.java | 2 +-
.../runners/dataflow/worker/apiary/Apiary.java | 2 +-
.../FixMultiOutputInfosOnParDoInstructions.java | 2 +-
.../runners/dataflow/worker/counters/Counter.java | 2 +-
.../dataflow/worker/counters/CounterFactory.java | 10 +-
.../dataflow/worker/counters/CounterName.java | 6 +-
.../dataflow/worker/counters/CounterSet.java | 4 +-
.../counters/DataflowCounterUpdateExtractor.java | 2 +-
.../worker/fn/control/BeamFnMapTaskExecutor.java | 6 +-
.../fn/control/ProcessRemoteBundleOperation.java | 2 +-
.../control/RegisterAndProcessBundleOperation.java | 12 +-
.../worker/fn/data/BeamFnDataGrpcService.java | 4 +-
.../fn/data/RemoteGrpcPortReadOperation.java | 2 +-
.../fn/data/RemoteGrpcPortWriteOperation.java | 4 +-
.../worker/fn/logging/BeamFnLoggingService.java | 2 +-
.../graph/CloneAmbiguousFlattensFunction.java | 6 +-
.../graph/CreateExecutableStageNodeFunction.java | 10 +-
.../graph/CreateRegisterFnOperationFunction.java | 14 +-
.../graph/DeduceFlattenLocationsFunction.java | 10 +-
.../worker/graph/DeduceNodeLocationsFunction.java | 4 +-
.../beam/runners/dataflow/worker/graph/Edges.java | 2 +-
...nsertFetchAndFilterStreamingSideInputNodes.java | 10 +-
.../worker/graph/LengthPrefixUnknownCoders.java | 8 +-
.../worker/graph/MapTaskToNetworkFunction.java | 6 +-
.../runners/dataflow/worker/graph/Networks.java | 18 +-
.../beam/runners/dataflow/worker/graph/Nodes.java | 4 +-
.../worker/graph/RegisterNodeFunction.java | 12 +-
.../graph/RemoveFlattenInstructionsFunction.java | 8 +-
.../graph/ReplacePgbkWithPrecombineFunction.java | 2 +-
.../logging/DataflowWorkerLoggingHandler.java | 6 +-
.../logging/DataflowWorkerLoggingInitializer.java | 6 +-
.../JulHandlerPrintStreamAdapterFactory.java | 2 +-
.../dataflow/worker/profiler/ScopedProfiler.java | 2 +-
.../dataflow/worker/status/BaseStatusServlet.java | 2 +-
.../dataflow/worker/status/DebugCapture.java | 2 +-
.../dataflow/worker/status/HeapzServlet.java | 2 +-
.../dataflow/worker/status/ThreadzServlet.java | 2 +-
.../dataflow/worker/status/WorkerStatusPages.java | 2 +-
.../util/BatchGroupAlsoByWindowAndCombineFn.java | 6 +-
.../util/BatchGroupAlsoByWindowViaIteratorsFn.java | 10 +-
.../worker/util/BatchGroupAlsoByWindowsDoFns.java | 2 +-
.../dataflow/worker/util/MemoryMonitor.java | 10 +-
.../dataflow/worker/util/ScalableBloomFilter.java | 14 +-
.../dataflow/worker/util/TimerOrElement.java | 4 +-
.../dataflow/worker/util/ValueInEmptyWindows.java | 2 +-
.../worker/util/common/ForwardingReiterator.java | 2 +-
.../common/worker/BatchingShuffleEntryReader.java | 4 +-
.../common/worker/ByteArrayShufflePosition.java | 6 +-
.../common/worker/CachingShuffleBatchReader.java | 12 +-
.../util/common/worker/ExecutionStateSampler.java | 6 +-
.../util/common/worker/ExecutionStateTracker.java | 6 +-
.../util/common/worker/FlattenOperation.java | 2 +-
.../worker/GroupingShuffleEntryIterator.java | 4 +-
.../common/worker/GroupingShuffleRangeTracker.java | 4 +-
.../worker/util/common/worker/GroupingTables.java | 2 +-
.../worker/util/common/worker/MapTaskExecutor.java | 4 +-
.../worker/util/common/worker/ParDoOperation.java | 2 +-
.../common/worker/ProgressTrackingReiterator.java | 2 +-
.../worker/util/common/worker/ReadOperation.java | 6 +-
.../util/common/worker/ShuffleReadCounter.java | 2 +-
.../worker/util/common/worker/WorkExecutor.java | 2 +-
.../util/common/worker/WorkProgressUpdater.java | 4 +-
.../worker/util/common/worker/WriteOperation.java | 2 +-
.../worker/windmill/GrpcWindmillServer.java | 53 +-
.../worker/windmill/WindmillServerBase.java | 2 +-
.../worker/windmill/WindmillServerStub.java | 8 +-
.../dataflow/harness/test/TestExecutors.java | 2 +-
.../dataflow/worker/BatchDataflowWorkerTest.java | 4 +-
.../worker/CombineValuesFnFactoryTest.java | 6 +-
...CreateIsmShardKeyAndSortKeyDoFnFactoryTest.java | 4 +-
.../DataflowElementExecutionTrackerTest.java | 2 +-
.../worker/DataflowOperationContextTest.java | 2 +-
.../worker/DataflowWorkUnitClientTest.java | 32 +-
.../dataflow/worker/DefaultParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/ExperimentContextTest.java | 2 +-
.../dataflow/worker/FakeWindmillServer.java | 4 +-
.../dataflow/worker/FnApiWindowMappingFnTest.java | 2 +-
.../GroupAlsoByWindowParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/GroupingShuffleReaderTest.java | 2 +-
.../IntrinsicMapTaskExecutorFactoryTest.java | 10 +-
.../worker/IntrinsicMapTaskExecutorTest.java | 2 +-
.../runners/dataflow/worker/IsmFormatTest.java | 2 +-
.../dataflow/worker/IsmReaderFactoryTest.java | 8 +-
.../runners/dataflow/worker/IsmReaderTest.java | 20 +-
.../dataflow/worker/IsmSideInputReaderTest.java | 32 +-
.../beam/runners/dataflow/worker/IsmSinkTest.java | 2 +-
.../LazilyInitializedSideInputReaderTest.java | 2 +-
.../dataflow/worker/LogRecordMatcherTest.java | 2 +-
.../beam/runners/dataflow/worker/LogSaverTest.java | 2 +-
.../runners/dataflow/worker/OrderedCodeTest.java | 4 +-
.../worker/PairWithConstantKeyDoFnFactoryTest.java | 2 +-
.../worker/PartialGroupByKeyParDoFnsTest.java | 6 +-
.../worker/PartitioningShuffleReaderTest.java | 2 +-
.../runners/dataflow/worker/ReaderCacheTest.java | 2 +-
...ReifyTimestampAndWindowsParDoFnFactoryTest.java | 4 +-
...essCoderCloudObjectTranslatorRegistrarTest.java | 4 +-
.../runners/dataflow/worker/ShuffleSinkTest.java | 2 +-
.../runners/dataflow/worker/SimpleParDoFnTest.java | 6 +-
.../runners/dataflow/worker/StateFetcherTest.java | 6 +-
.../worker/StreamingDataflowWorkerTest.java | 12 +-
...eamingKeyedWorkItemSideInputDoFnRunnerTest.java | 4 +-
.../worker/StreamingModeExecutionContextTest.java | 2 +-
...eamingPCollectionViewWriterDoFnFactoryTest.java | 2 +-
.../worker/StreamingSideInputDoFnRunnerTest.java | 2 +-
.../worker/StreamingSideInputFetcherTest.java | 4 +-
.../runners/dataflow/worker/TestShuffleReader.java | 2 +-
.../ToIsmRecordForMultimapDoFnFactoryTest.java | 4 +-
.../worker/UngroupedShuffleReaderTest.java | 2 +-
.../dataflow/worker/UserParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/ValuesDoFnFactoryTest.java | 2 +-
.../worker/WindmillStateInternalsTest.java | 8 +-
.../worker/WindmillTimerInternalsTest.java | 2 +-
.../dataflow/worker/WorkItemStatusClientTest.java | 2 +-
.../WorkerCustomSourcesSplitOnlySourceTest.java | 2 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 8 +-
.../worker/WorkerPipelineOptionsFactoryTest.java | 2 +-
...FixMultiOutputInfosOnParDoInstructionsTest.java | 2 +-
.../worker/counters/CounterFactoryTest.java | 2 +-
.../dataflow/worker/counters/CounterTest.java | 2 +-
.../worker/fn/BeamFnControlServiceTest.java | 4 +-
.../fn/control/BeamFnMapTaskExecutorTest.java | 8 +-
.../RegisterAndProcessBundleOperationTest.java | 6 +-
.../fn/logging/BeamFnLoggingServiceTest.java | 2 +-
.../graph/CloneAmbiguousFlattensFunctionTest.java | 8 +-
.../CreateRegisterFnOperationFunctionTest.java | 12 +-
.../graph/DeduceFlattenLocationsFunctionTest.java | 10 +-
.../graph/DeduceNodeLocationsFunctionTest.java | 14 +-
...tFetchAndFilterStreamingSideInputNodesTest.java | 16 +-
.../graph/LengthPrefixUnknownCodersTest.java | 8 +-
.../worker/graph/MapTaskToNetworkFunctionTest.java | 6 +-
.../dataflow/worker/graph/NetworksTest.java | 10 +-
.../runners/dataflow/worker/graph/NodesTest.java | 4 +-
.../RemoveFlattenInstructionsFunctionTest.java | 10 +-
.../ReplacePgbkWithPrecombineFunctionTest.java | 8 +-
.../logging/DataflowWorkerLoggingHandlerTest.java | 2 +-
.../DataflowWorkerLoggingInitializerTest.java | 2 +-
.../JulHandlerPrintStreamAdapterFactoryTest.java | 2 +-
.../worker/profiler/ScopedProfilerTest.java | 2 +-
.../dataflow/worker/status/ThreadzServletTest.java | 2 +-
.../worker/util/CounterHamcrestMatchers.java | 4 +-
.../util/DataflowCounterUpdateExtractorTest.java | 2 +-
.../worker/util/GroupAlsoByWindowProperties.java | 10 +-
.../dataflow/worker/util/ListOutputManager.java | 4 +-
.../common/worker/ExecutionStateSamplerTest.java | 2 +-
.../util/common/worker/ExecutorTestUtils.java | 2 +-
.../worker/GroupingShuffleEntryIteratorTest.java | 4 +-
.../util/common/worker/MapTaskExecutorTest.java | 2 +-
.../util/common/worker/ReadOperationTest.java | 2 +-
.../util/common/worker/TestOutputReceiver.java | 2 +-
runners/java-fn-execution/build.gradle | 4 +-
.../beam/runners/fnexecution/GrpcFnServer.java | 2 +-
.../beam/runners/fnexecution/ServerFactory.java | 4 +-
.../BeamFileSystemArtifactRetrievalService.java | 18 +-
.../BeamFileSystemArtifactStagingService.java | 6 +-
.../control/DefaultJobBundleFactory.java | 16 +-
.../control/FnApiControlClientPoolService.java | 4 +-
.../fnexecution/control/MapControlClientPool.java | 2 +-
.../control/ProcessBundleDescriptors.java | 10 +-
.../fnexecution/control/SdkHarnessClient.java | 6 +-
.../SingleEnvironmentInstanceJobBundleFactory.java | 2 +-
.../runners/fnexecution/data/GrpcDataService.java | 2 +-
.../fnexecution/environment/DockerCommand.java | 24 +-
.../environment/DockerEnvironmentFactory.java | 12 +-
.../environment/EmbeddedEnvironmentFactory.java | 11 +-
.../environment/ExternalEnvironmentFactory.java | 2 +-
.../environment/ProcessEnvironmentFactory.java | 4 +-
.../fnexecution/environment/ProcessManager.java | 6 +-
.../fnexecution/logging/GrpcLoggingService.java | 2 +-
.../splittabledofn/SDFFeederViaStateAndTimers.java | 6 +-
.../fnexecution/state/GrpcStateService.java | 2 +-
.../fnexecution/state/StateRequestHandlers.java | 4 +-
.../beam/runners/fnexecution/wire/WireCoders.java | 2 +-
.../runners/fnexecution/EmbeddedSdkHarness.java | 2 +-
.../GrpcContextHeaderAccessorProviderTest.java | 2 +-
.../runners/fnexecution/ServerFactoryTest.java | 6 +-
.../BeamFileSystemArtifactServicesTest.java | 10 +-
.../control/DefaultJobBundleFactoryTest.java | 2 +-
.../fnexecution/control/RemoteExecutionTest.java | 18 +-
.../fnexecution/control/SdkHarnessClientTest.java | 6 +-
...gleEnvironmentInstanceJobBundleFactoryTest.java | 2 +-
.../fnexecution/environment/DockerCommandTest.java | 6 +-
.../environment/DockerEnvironmentFactoryTest.java | 4 +
.../wire/LengthPrefixUnknownCodersTest.java | 2 +-
runners/reference/java/build.gradle | 4 +-
.../beam/runners/reference/CloseableResource.java | 4 +-
.../beam/runners/reference/PortableRunner.java | 8 +-
.../runners/reference/PortableRunnerRegistrar.java | 2 +-
.../testing/TestPortablePipelineOptions.java | 2 +-
runners/reference/job-server/build.gradle | 2 -
runners/samza/build.gradle | 2 +-
.../samza/SamzaPipelineOptionsValidator.java | 2 +-
.../beam/runners/samza/SamzaRunnerRegistrar.java | 2 +-
.../runners/samza/adapter/BoundedSourceSystem.java | 4 +-
.../samza/adapter/UnboundedSourceSystem.java | 6 +-
.../apache/beam/runners/samza/runtime/DoFnOp.java | 4 +-
.../runtime/DoFnRunnerWithKeyedInternals.java | 2 +-
.../beam/runners/samza/runtime/GroupByKeyOp.java | 2 +-
.../beam/runners/samza/runtime/KeyedInternals.java | 2 +-
.../runners/samza/runtime/SamzaAssignContext.java | 2 +-
.../samza/runtime/SamzaStoreStateInternals.java | 8 +-
.../runners/samza/translation/ConfigBuilder.java | 8 +-
.../runners/samza/translation/ConfigContext.java | 2 +-
.../translation/ParDoBoundMultiTranslator.java | 2 +-
.../runners/samza/translation/ReadTranslator.java | 2 +-
.../samza/translation/SamzaPipelineTranslator.java | 4 +-
.../SamzaPublishViewTransformOverride.java | 2 +-
.../samza/translation/SamzaTransformOverrides.java | 2 +-
.../samza/translation/TranslationContext.java | 2 +-
.../samza/adapter/BoundedSourceSystemTest.java | 2 +-
.../runtime/SamzaStoreStateInternalsTest.java | 6 +-
.../runners/spark/SparkNativePipelineVisitor.java | 4 +-
.../org/apache/beam/runners/spark/SparkRunner.java | 6 +-
.../beam/runners/spark/SparkRunnerRegistrar.java | 2 +-
.../runners/spark/SparkTransformOverrides.java | 2 +-
.../apache/beam/runners/spark/TestSparkRunner.java | 4 +-
.../spark/aggregators/AggregatorsAccumulator.java | 4 +-
.../spark/aggregators/NamedAggregators.java | 4 +-
.../spark/coders/BeamSparkRunnerRegistrator.java | 43 +-
.../beam/runners/spark/coders/CoderHelpers.java | 2 +-
.../apache/beam/runners/spark/io/CreateStream.java | 4 +-
.../beam/runners/spark/io/MicrobatchSource.java | 12 +-
.../beam/runners/spark/io/SourceDStream.java | 2 +-
.../apache/beam/runners/spark/io/SourceRDD.java | 6 +-
.../runners/spark/io/SparkUnboundedSource.java | 2 +-
.../runners/spark/metrics/MetricsAccumulator.java | 4 +-
.../runners/spark/metrics/SparkBeamMetric.java | 2 +-
.../runners/spark/metrics/WithMetricsSupport.java | 16 +-
.../SparkGroupAlsoByWindowViaWindowSet.java | 15 +-
.../spark/stateful/SparkStateInternals.java | 4 +-
.../spark/stateful/SparkTimerInternals.java | 6 +-
.../runners/spark/stateful/StateSpecFunctions.java | 6 +-
.../spark/translation/EvaluationContext.java | 4 +-
.../spark/translation/GroupCombineFunctions.java | 251 ++++++----
.../spark/translation/MultiDoFnFunction.java | 8 +-
.../spark/translation/SparkAbstractCombineFn.java | 8 +-
.../spark/translation/SparkAssignWindowFn.java | 2 +-
.../spark/translation/SparkGlobalCombineFn.java | 4 +-
.../spark/translation/SparkKeyedCombineFn.java | 4 +-
.../spark/translation/SparkProcessContext.java | 4 +-
.../spark/translation/TransformTranslator.java | 20 +-
.../spark/translation/TranslationUtils.java | 6 +-
.../SparkRunnerStreamingContextFactory.java | 2 +-
.../streaming/StreamingTransformTranslator.java | 6 +-
.../streaming/WatermarkSyncedDStream.java | 6 +-
.../apache/beam/runners/spark/util/ByteArray.java | 2 +-
.../runners/spark/util/GlobalWatermarkHolder.java | 12 +-
.../runners/spark/util/SparkSideInputReader.java | 4 +-
.../runners/spark/ProvidedSparkContextTest.java | 4 +-
.../runners/spark/SparkRunnerRegistrarTest.java | 4 +-
.../spark/TestSparkPipelineOptionsRegistrar.java | 2 +-
.../aggregators/metrics/sink/InMemoryMetrics.java | 2 +-
.../metrics/sink/SparkMetricsSinkTest.java | 4 +-
.../coders/BeamSparkRunnerRegistratorTest.java | 109 +++++
.../beam/runners/spark/io/AvroPipelineTest.java | 4 +-
.../beam/runners/spark/io/NumShardsTest.java | 4 +-
.../spark/io/ReaderToIteratorAdapterTest.java | 2 +-
.../spark/metrics/SparkMetricsPusherTest.java | 10 +-
.../translation/GroupCombineFunctionsTest.java | 131 +++++
.../ResumeFromCheckpointStreamingTest.java | 10 +-
.../streaming/SparkCoGroupByKeyStreamingTest.java | 2 +-
sdks/go/cmd/starcgen/starcgen.go | 99 ++--
sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go | 6 +-
sdks/go/pkg/beam/artifact/gcsproxy/staging.go | 7 +-
sdks/go/pkg/beam/beam.shims.go | 336 +++++++++++++
sdks/go/pkg/beam/coder.go | 7 -
.../pkg/beam/core/runtime/coderx/coderx.shims.go | 320 ++++++++++++
sdks/go/pkg/beam/core/runtime/coderx/float.go | 6 -
sdks/go/pkg/beam/core/runtime/coderx/int.go | 13 +-
sdks/go/pkg/beam/core/runtime/coderx/varint.go | 9 +-
sdks/go/pkg/beam/create.go | 4 -
sdks/go/pkg/beam/partition.go | 4 -
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 2 +-
.../pkg/beam/runners/dataflow/dataflowlib/stage.go | 2 +-
sdks/go/pkg/beam/testing/passert/hash.go | 5 -
sdks/go/pkg/beam/testing/passert/passert.go | 9 +-
sdks/go/pkg/beam/testing/passert/passert.shims.go | 405 ++++++++++++++++
sdks/go/pkg/beam/testing/passert/sum.go | 5 -
sdks/go/pkg/beam/transforms/filter/distinct.go | 5 -
sdks/go/pkg/beam/transforms/filter/filter.go | 9 +-
sdks/go/pkg/beam/transforms/filter/filter.shims.go | 250 ++++++++++
sdks/go/pkg/beam/transforms/stats/count.go | 6 +-
sdks/go/pkg/beam/transforms/stats/stats.shims.go | 13 +-
sdks/go/pkg/beam/transforms/stats/util_gen.go | 7 +-
sdks/go/pkg/beam/transforms/stats/util_gen.tmpl | 9 +-
sdks/go/pkg/beam/transforms/top/top.go | 8 +-
sdks/go/pkg/beam/transforms/top/top.shims.go | 156 ++++++
sdks/go/pkg/beam/util.go | 9 +-
sdks/go/pkg/beam/util/gcsx/gcs.go | 20 +-
sdks/go/pkg/beam/util/gcsx/gcs_test.go | 6 +-
sdks/go/pkg/beam/util/shimx/generate.go | 8 +
sdks/go/pkg/beam/util/starcgenx/starcgenx.go | 8 +-
sdks/go/pkg/beam/x/debug/debug.shims.go | 394 +++++++++++++++
sdks/go/pkg/beam/x/debug/head.go | 8 +-
sdks/go/pkg/beam/x/debug/print.go | 8 -
.../src/main/resources/beam/checkstyle.xml | 21 +
.../src/main/resources/beam/findbugs-filter.xml | 7 +
.../src/main/resources/beam/suppressions.xml | 5 +-
sdks/java/core/build.gradle | 2 +-
.../main/java/org/apache/beam/sdk/Pipeline.java | 28 +-
.../java/org/apache/beam/sdk/PipelineRunner.java | 2 +-
.../apache/beam/sdk/annotations/Experimental.java | 4 +
.../java/org/apache/beam/sdk/coders/AvroCoder.java | 4 +-
.../apache/beam/sdk/coders/BigDecimalCoder.java | 2 +-
.../apache/beam/sdk/coders/BigIntegerCoder.java | 2 +-
.../org/apache/beam/sdk/coders/ByteArrayCoder.java | 2 +-
.../java/org/apache/beam/sdk/coders/Coder.java | 12 +-
.../org/apache/beam/sdk/coders/CoderProviders.java | 4 +-
.../org/apache/beam/sdk/coders/CoderRegistry.java | 22 +-
.../org/apache/beam/sdk/coders/DefaultCoder.java | 2 +-
.../org/apache/beam/sdk/coders/DelegateCoder.java | 4 +-
.../apache/beam/sdk/coders/IterableLikeCoder.java | 2 +-
.../apache/beam/sdk/coders/LengthPrefixCoder.java | 6 +-
.../java/org/apache/beam/sdk/coders/MapCoder.java | 2 +-
.../org/apache/beam/sdk/coders/NullableCoder.java | 4 +-
.../java/org/apache/beam/sdk/coders/RowCoder.java | 4 +-
.../apache/beam/sdk/coders/RowCoderGenerator.java | 4 +-
.../apache/beam/sdk/coders/SerializableCoder.java | 2 +-
.../apache/beam/sdk/coders/ShardedKeyCoder.java | 2 +-
.../org/apache/beam/sdk/coders/SnappyCoder.java | 2 +-
.../apache/beam/sdk/coders/StringUtf8Coder.java | 4 +-
.../beam/sdk/coders/StructuralByteArray.java | 2 +-
.../main/java/org/apache/beam/sdk/io/AvroIO.java | 14 +-
.../java/org/apache/beam/sdk/io/AvroSource.java | 10 +-
.../java/org/apache/beam/sdk/io/AvroUtils.java | 6 +-
.../sdk/io/BoundedReadFromUnboundedSource.java | 2 +-
.../org/apache/beam/sdk/io/CompressedSource.java | 4 +-
.../java/org/apache/beam/sdk/io/Compression.java | 7 +-
.../beam/sdk/io/ConstantAvroDestination.java | 8 +-
.../org/apache/beam/sdk/io/CountingSource.java | 6 +-
.../apache/beam/sdk/io/DefaultFilenamePolicy.java | 8 +-
.../beam/sdk/io/DynamicAvroDestinations.java | 2 +-
.../beam/sdk/io/DynamicFileDestinations.java | 2 +-
.../java/org/apache/beam/sdk/io/FileBasedSink.java | 26 +-
.../org/apache/beam/sdk/io/FileBasedSource.java | 10 +-
.../main/java/org/apache/beam/sdk/io/FileIO.java | 12 +-
.../java/org/apache/beam/sdk/io/FileSystems.java | 30 +-
.../org/apache/beam/sdk/io/GenerateSequence.java | 2 +-
.../org/apache/beam/sdk/io/LocalFileSystem.java | 10 +-
.../beam/sdk/io/LocalFileSystemRegistrar.java | 2 +-
.../org/apache/beam/sdk/io/LocalResourceId.java | 6 +-
.../org/apache/beam/sdk/io/OffsetBasedSource.java | 2 +-
.../beam/sdk/io/SerializableAvroCodecFactory.java | 4 +-
.../java/org/apache/beam/sdk/io/TFRecordIO.java | 14 +-
.../main/java/org/apache/beam/sdk/io/TextIO.java | 14 +-
.../java/org/apache/beam/sdk/io/TextSource.java | 4 +-
.../java/org/apache/beam/sdk/io/WriteFiles.java | 16 +-
.../org/apache/beam/sdk/io/WriteFilesResult.java | 2 +-
.../apache/beam/sdk/io/fs/ResourceIdTester.java | 8 +-
.../java/org/apache/beam/sdk/io/range/ByteKey.java | 2 +-
.../org/apache/beam/sdk/io/range/ByteKeyRange.java | 12 +-
.../beam/sdk/io/range/ByteKeyRangeTracker.java | 4 +-
.../org/apache/beam/sdk/io/range/OffsetRange.java | 2 +-
.../beam/sdk/io/range/OffsetRangeTracker.java | 4 +-
.../org/apache/beam/sdk/metrics/MetricName.java | 4 +-
.../apache/beam/sdk/metrics/MetricNameFilter.java | 2 +-
.../org/apache/beam/sdk/metrics/MetricsFilter.java | 2 +-
.../apache/beam/sdk/metrics/MetricsOptions.java | 83 ++++
.../org/apache/beam/sdk/metrics/SourceMetrics.java | 2 +-
.../options/DefaultPipelineOptionsRegistrar.java | 2 +-
.../options/ManualDockerEnvironmentOptions.java | 2 +-
.../apache/beam/sdk/options/PipelineOptions.java | 57 +--
.../beam/sdk/options/PipelineOptionsFactory.java | 50 +-
.../beam/sdk/options/PipelineOptionsReflector.java | 6 +-
.../beam/sdk/options/PipelineOptionsValidator.java | 12 +-
.../beam/sdk/options/ProxyInvocationHandler.java | 22 +-
.../apache/beam/sdk/options/SdkHarnessOptions.java | 2 +-
.../org/apache/beam/sdk/options/ValueProvider.java | 4 +-
.../apache/beam/sdk/options/ValueProviders.java | 2 +-
.../beam/sdk/runners/TransformHierarchy.java | 16 +-
.../apache/beam/sdk/schemas/AutoValueSchema.java | 113 +++++
.../beam/sdk/schemas/FieldAccessDescriptor.java | 8 +-
.../beam/sdk/schemas/FieldTypeDescriptors.java | 6 +-
.../sdk/schemas/FieldValueTypeInformation.java | 2 +-
.../beam/sdk/schemas/FromRowUsingCreator.java | 16 +-
.../apache/beam/sdk/schemas/JavaBeanSchema.java | 90 ++--
.../apache/beam/sdk/schemas/JavaFieldSchema.java | 78 ++-
.../java/org/apache/beam/sdk/schemas/Schema.java | 8 +-
.../apache/beam/sdk/schemas/SchemaRegistry.java | 9 +-
.../schemas/SchemaUserTypeConstructorCreator.java | 2 +-
.../schemas/{ => annotations}/DefaultSchema.java | 13 +-
.../beam/sdk/schemas/annotations/SchemaCreate.java | 74 +++
.../sdk/schemas/annotations/SchemaFieldName.java | 54 +++
.../beam/sdk/schemas/annotations/SchemaIgnore.java | 51 ++
.../sdk/schemas/annotations/package-info.java} | 24 +-
.../apache/beam/sdk/schemas/transforms/Cast.java | 10 +-
.../beam/sdk/schemas/transforms/CoGroup.java | 6 +-
.../apache/beam/sdk/schemas/transforms/Filter.java | 4 +-
.../sdk/schemas/transforms/SchemaAggregateFn.java | 2 +-
.../apache/beam/sdk/schemas/transforms/Unnest.java | 2 +-
.../beam/sdk/schemas/utils/AutoValueUtils.java | 294 +++++++++++
.../beam/sdk/schemas/utils/AvroByteBuddyUtils.java | 2 +-
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 36 +-
.../beam/sdk/schemas/utils/ByteBuddyUtils.java | 180 ++++++-
.../sdk/schemas/utils/FieldValueTypeSupplier.java | 11 +-
.../beam/sdk/schemas/utils/JavaBeanUtils.java | 126 ++++-
.../apache/beam/sdk/schemas/utils/POJOUtils.java | 104 +++-
.../beam/sdk/schemas/utils/ReflectUtils.java | 68 ++-
.../beam/sdk/schemas/utils/SchemaZipFold.java | 2 +-
.../sdk/schemas/utils/StaticSchemaInference.java | 36 +-
.../java/org/apache/beam/sdk/state/StateSpecs.java | 2 +-
.../org/apache/beam/sdk/testing/Annotations.java | 4 +-
.../apache/beam/sdk/testing/CoderProperties.java | 12 +-
.../beam/sdk/testing/FileChecksumMatcher.java | 10 +-
.../beam/sdk/testing/MatcherDeserializer.java | 2 +-
.../apache/beam/sdk/testing/MatcherSerializer.java | 2 +-
.../java/org/apache/beam/sdk/testing/PAssert.java | 8 +-
.../beam/sdk/testing/SerializableMatchers.java | 2 +-
.../apache/beam/sdk/testing/SourceTestUtils.java | 6 +-
.../org/apache/beam/sdk/testing/StaticWindows.java | 6 +-
.../apache/beam/sdk/testing/SuccessOrFailure.java | 4 +-
.../org/apache/beam/sdk/testing/TestPipeline.java | 14 +-
.../org/apache/beam/sdk/testing/TestStream.java | 6 +-
.../apache/beam/sdk/testing/WindowFnTestUtils.java | 6 +-
.../apache/beam/sdk/testing/WindowSupplier.java | 6 +-
.../beam/sdk/transforms/ApproximateQuantiles.java | 8 +-
.../beam/sdk/transforms/ApproximateUnique.java | 12 +-
.../org/apache/beam/sdk/transforms/Combine.java | 6 +-
.../apache/beam/sdk/transforms/CombineFnBase.java | 2 +-
.../org/apache/beam/sdk/transforms/CombineFns.java | 16 +-
.../org/apache/beam/sdk/transforms/Contextful.java | 2 +-
.../org/apache/beam/sdk/transforms/Create.java | 12 +-
.../beam/sdk/transforms/DoFnOutputReceivers.java | 4 +-
.../org/apache/beam/sdk/transforms/DoFnTester.java | 8 +-
.../beam/sdk/transforms/FlatMapElements.java | 4 +-
.../beam/sdk/transforms/GroupIntoBatches.java | 6 +-
.../org/apache/beam/sdk/transforms/Latest.java | 8 +-
.../apache/beam/sdk/transforms/MapElements.java | 4 +-
.../java/org/apache/beam/sdk/transforms/Mean.java | 2 +-
.../java/org/apache/beam/sdk/transforms/ParDo.java | 4 +-
.../apache/beam/sdk/transforms/Requirements.java | 2 +-
.../org/apache/beam/sdk/transforms/Sample.java | 2 +-
.../org/apache/beam/sdk/transforms/ToString.java | 2 +-
.../java/org/apache/beam/sdk/transforms/Top.java | 4 +-
.../java/org/apache/beam/sdk/transforms/Wait.java | 4 +-
.../java/org/apache/beam/sdk/transforms/Watch.java | 24 +-
.../org/apache/beam/sdk/transforms/WithKeys.java | 2 +-
.../apache/beam/sdk/transforms/WithTimestamps.java | 2 +-
.../beam/sdk/transforms/display/DisplayData.java | 14 +-
.../beam/sdk/transforms/join/CoGbkResult.java | 8 +-
.../sdk/transforms/join/KeyedPCollectionTuple.java | 2 +-
.../reflect/ByteBuddyDoFnInvokerFactory.java | 2 +-
.../reflect/ByteBuddyOnTimerInvokerFactory.java | 12 +-
.../beam/sdk/transforms/reflect/DoFnSignature.java | 2 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 12 +-
.../reflect/StableInvokerNamingStrategy.java | 2 +-
.../splittabledofn/ByteKeyRangeTracker.java | 12 +-
.../splittabledofn/OffsetRangeTracker.java | 8 +-
.../beam/sdk/transforms/windowing/AfterAll.java | 4 +-
.../beam/sdk/transforms/windowing/AfterEach.java | 4 +-
.../beam/sdk/transforms/windowing/AfterFirst.java | 4 +-
.../transforms/windowing/AfterProcessingTime.java | 2 +-
.../windowing/AfterSynchronizedProcessingTime.java | 2 +-
.../sdk/transforms/windowing/AfterWatermark.java | 4 +-
.../sdk/transforms/windowing/OrFinallyTrigger.java | 2 +-
.../beam/sdk/transforms/windowing/PaneInfo.java | 8 +-
.../transforms/windowing/TimestampCombiner.java | 6 +-
.../beam/sdk/transforms/windowing/Trigger.java | 2 +-
.../beam/sdk/transforms/windowing/Window.java | 4 +-
.../java/org/apache/beam/sdk/util/ApiSurface.java | 32 +-
.../org/apache/beam/sdk/util/AppliedCombineFn.java | 2 +-
.../apache/beam/sdk/util/BucketingFunction.java | 2 +-
.../util/BufferedElementCountingOutputStream.java | 2 +-
.../java/org/apache/beam/sdk/util/ClassPath.java | 38 +-
.../java/org/apache/beam/sdk/util/CoderUtils.java | 4 +-
.../apache/beam/sdk/util/ExplicitShardedFile.java | 8 +-
.../sdk/util/FilePatternMatchingShardedFile.java | 10 +-
.../org/apache/beam/sdk/util/FluentBackoff.java | 4 +-
.../org/apache/beam/sdk/util/InstanceBuilder.java | 6 +-
.../org/apache/beam/sdk/util/MovingFunction.java | 2 +-
.../java/org/apache/beam/sdk/util/NameUtils.java | 6 +-
.../apache/beam/sdk/util/NumberedShardedFile.java | 16 +-
.../java/org/apache/beam/sdk/util/ReleaseInfo.java | 2 +-
.../apache/beam/sdk/util/RowJsonDeserializer.java | 2 +-
.../apache/beam/sdk/util/RowJsonValidation.java | 2 +-
.../apache/beam/sdk/util/SerializableUtils.java | 2 +-
.../java/org/apache/beam/sdk/util/StringUtils.java | 2 +-
.../apache/beam/sdk/util/UnownedInputStream.java | 2 +-
.../apache/beam/sdk/util/UnownedOutputStream.java | 2 +-
.../org/apache/beam/sdk/util/WindowedValue.java | 8 +-
.../java/org/apache/beam/sdk/util/ZipFiles.java | 16 +-
.../beam/sdk/util/common/ReflectHelpers.java | 14 +-
.../main/java/org/apache/beam/sdk/values/KV.java | 2 +-
.../org/apache/beam/sdk/values/PCollection.java | 4 +-
.../apache/beam/sdk/values/PCollectionList.java | 4 +-
.../apache/beam/sdk/values/PCollectionTuple.java | 2 +-
.../apache/beam/sdk/values/PCollectionViews.java | 10 +-
.../org/apache/beam/sdk/values/PValueBase.java | 2 +-
.../main/java/org/apache/beam/sdk/values/Row.java | 10 +-
.../org/apache/beam/sdk/values/RowWithGetters.java | 4 +-
.../org/apache/beam/sdk/values/TaggedPValue.java | 2 +-
.../apache/beam/sdk/values/TimestampedValue.java | 2 +-
.../java/org/apache/beam/sdk/values/TupleTag.java | 4 +-
.../org/apache/beam/sdk/values/TupleTagList.java | 4 +-
.../org/apache/beam/sdk/values/TypeDescriptor.java | 10 +-
.../org/apache/beam/sdk/values/TypeParameter.java | 2 +-
.../beam/sdk/values/ValueInSingleWindow.java | 2 +-
.../apache/beam/sdk/values/ValueWithRecordId.java | 2 +-
.../apache/beam/sdk/values/WindowingStrategy.java | 4 +-
.../org/apache/beam/SdkCoreApiSurfaceTest.java | 2 +-
.../java/org/apache/beam/sdk/PipelineTest.java | 4 +-
.../apache/beam/sdk/coders/AvroCoderTestPojo.java | 2 +-
.../beam/sdk/coders/BigDecimalCoderTest.java | 2 +-
.../beam/sdk/coders/BigIntegerCoderTest.java | 2 +-
.../apache/beam/sdk/coders/CoderRegistryTest.java | 2 +-
.../apache/beam/sdk/coders/DefaultCoderTest.java | 2 +-
.../apache/beam/sdk/coders/DelegateCoderTest.java | 4 +-
.../apache/beam/sdk/coders/DurationCoderTest.java | 2 +-
.../apache/beam/sdk/coders/InstantCoderTest.java | 4 +-
.../beam/sdk/coders/LengthPrefixCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/MapCoderTest.java | 2 +-
.../apache/beam/sdk/coders/NullableCoderTest.java | 4 +-
.../beam/sdk/coders/PrintBase64Encodings.java | 4 +-
.../beam/sdk/coders/StructuralByteArrayTest.java | 2 +-
.../beam/sdk/coders/StructuredCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/RowCoderTest.java | 2 +-
.../java/org/apache/beam/sdk/io/AvroIOTest.java | 24 +-
.../org/apache/beam/sdk/io/AvroSourceTest.java | 2 +-
.../sdk/io/BoundedReadFromUnboundedSourceTest.java | 4 +-
.../apache/beam/sdk/io/CompressedSourceTest.java | 10 +-
.../org/apache/beam/sdk/io/FileBasedSinkTest.java | 4 +-
.../java/org/apache/beam/sdk/io/FileIOTest.java | 4 +-
.../org/apache/beam/sdk/io/FileSystemsTest.java | 8 +-
.../beam/sdk/io/LocalFileSystemRegistrarTest.java | 2 +-
.../apache/beam/sdk/io/LocalFileSystemTest.java | 10 +-
.../apache/beam/sdk/io/LocalResourceIdTest.java | 33 +-
.../org/apache/beam/sdk/io/TFRecordIOTest.java | 19 +-
.../org/apache/beam/sdk/io/TextIOReadTest.java | 12 +-
.../org/apache/beam/sdk/io/TextIOWriteTest.java | 20 +-
.../org/apache/beam/sdk/io/WriteFilesTest.java | 8 +-
.../io/range/ByteKeyRangeEstimateFractionTest.java | 2 +-
.../io/range/ByteKeyRangeInterpolateKeyTest.java | 2 +-
.../apache/beam/sdk/io/range/ByteKeyRangeTest.java | 2 +-
.../sdk/options/PipelineOptionsFactoryTest.java | 10 +-
.../sdk/options/PipelineOptionsReflectorTest.java | 2 +-
.../beam/sdk/options/PipelineOptionsTest.java | 2 +-
.../sdk/options/ProxyInvocationHandlerTest.java | 8 +-
.../beam/sdk/options/SdkHarnessOptionsTest.java | 2 +-
.../apache/beam/sdk/options/ValueProviderTest.java | 2 +-
.../beam/sdk/options/ValueProvidersTest.java | 2 +-
.../beam/sdk/runners/TransformHierarchyTest.java | 2 +-
.../beam/sdk/schemas/AutoValueSchemaTest.java | 500 +++++++++++++++++++
.../apache/beam/sdk/schemas/AvroSchemaTest.java | 4 +-
.../sdk/schemas/FieldAccessDescriptorTest.java | 2 +-
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 72 ++-
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 78 ++-
.../beam/sdk/schemas/SchemaRegistryTest.java | 9 +-
.../beam/sdk/schemas/transforms/CastTest.java | 538 +++++++--------------
.../sdk/schemas/transforms/CastValidatorTest.java | 4 +-
.../beam/sdk/schemas/transforms/CoGroupTest.java | 4 +-
.../beam/sdk/schemas/transforms/ConvertTest.java | 6 +-
.../beam/sdk/schemas/transforms/FilterTest.java | 4 +-
.../beam/sdk/schemas/transforms/GroupTest.java | 6 +-
.../beam/sdk/schemas/transforms/SelectTest.java | 2 +-
.../beam/sdk/schemas/utils/AvroGenerators.java | 6 +-
.../beam/sdk/schemas/utils/AvroUtilsTest.java | 8 +-
.../beam/sdk/schemas/utils/JavaBeanUtilsTest.java | 38 +-
.../beam/sdk/schemas/utils/POJOUtilsTest.java | 36 +-
.../beam/sdk/schemas/utils/SchemaZipFoldTest.java | 2 +-
.../beam/sdk/schemas/utils/TestJavaBeans.java | 146 +++++-
.../apache/beam/sdk/schemas/utils/TestPOJOs.java | 248 +++++++++-
.../beam/sdk/testing/CoderPropertiesTest.java | 2 +-
.../beam/sdk/testing/CombineFnTesterTest.java | 2 +-
.../beam/sdk/testing/FileChecksumMatcherTest.java | 2 +-
.../beam/sdk/testing/GatherAllPanesTest.java | 2 +-
.../sdk/testing/InterceptingUrlClassLoader.java | 6 +-
.../org/apache/beam/sdk/testing/PAssertTest.java | 25 +-
.../beam/sdk/testing/PCollectionViewTesting.java | 2 +-
.../beam/sdk/testing/PaneExtractorsTest.java | 2 +-
.../beam/sdk/testing/SerializableMatchersTest.java | 2 +-
.../beam/sdk/testing/SourceTestUtilsTest.java | 2 +-
.../apache/beam/sdk/testing/StaticWindowsTest.java | 2 +-
.../beam/sdk/testing/WindowSupplierTest.java | 2 +-
.../sdk/transforms/ApproximateQuantilesTest.java | 2 +-
.../beam/sdk/transforms/ApproximateUniqueTest.java | 4 +-
.../apache/beam/sdk/transforms/CombineFnsTest.java | 4 +-
.../apache/beam/sdk/transforms/CombineTest.java | 12 +-
.../org/apache/beam/sdk/transforms/CreateTest.java | 4 +-
.../apache/beam/sdk/transforms/DistinctTest.java | 4 +-
.../apache/beam/sdk/transforms/DoFnTesterTest.java | 2 +-
.../beam/sdk/transforms/FlatMapElementsTest.java | 4 +-
.../apache/beam/sdk/transforms/FlattenTest.java | 2 +-
.../apache/beam/sdk/transforms/GroupByKeyTest.java | 2 +-
.../beam/sdk/transforms/GroupIntoBatchesTest.java | 2 +-
.../apache/beam/sdk/transforms/LatestFnTest.java | 2 +-
.../org/apache/beam/sdk/transforms/MaxTest.java | 2 +-
.../org/apache/beam/sdk/transforms/MeanTest.java | 2 +-
.../org/apache/beam/sdk/transforms/MinTest.java | 2 +-
.../beam/sdk/transforms/ParDoSchemaTest.java | 4 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 14 +-
.../apache/beam/sdk/transforms/ReshuffleTest.java | 2 +-
.../org/apache/beam/sdk/transforms/SampleTest.java | 6 +-
.../beam/sdk/transforms/SplittableDoFnTest.java | 11 +-
.../org/apache/beam/sdk/transforms/SumTest.java | 2 +-
.../org/apache/beam/sdk/transforms/ViewTest.java | 4 +-
.../org/apache/beam/sdk/transforms/WaitTest.java | 4 +-
.../org/apache/beam/sdk/transforms/WatchTest.java | 19 +-
.../transforms/display/DisplayDataEvaluator.java | 2 +-
.../transforms/display/DisplayDataMatchers.java | 2 +-
.../sdk/transforms/display/DisplayDataTest.java | 8 +-
.../sdk/transforms/join/CoGbkResultCoderTest.java | 2 +-
.../beam/sdk/transforms/join/CoGroupByKeyTest.java | 4 +-
.../beam/sdk/transforms/join/UnionCoderTest.java | 2 +-
.../reflect/DoFnSignaturesSplittableDoFnTest.java | 2 +-
.../sdk/transforms/windowing/GlobalWindowTest.java | 4 +-
.../transforms/windowing/IntervalWindowTest.java | 2 +-
.../sdk/transforms/windowing/SessionsTest.java | 2 +-
.../beam/sdk/transforms/windowing/StubTrigger.java | 2 +-
.../beam/sdk/transforms/windowing/WindowTest.java | 2 +-
.../sdk/transforms/windowing/WindowingTest.java | 2 +-
.../org/apache/beam/sdk/util/ApiSurfaceTest.java | 8 +-
.../BufferedElementCountingOutputStreamTest.java | 4 +-
.../apache/beam/sdk/util/CombineFnUtilTest.java | 2 +-
.../sdk/util/ExposedByteArrayInputStreamTest.java | 2 +-
.../sdk/util/ExposedByteArrayOutputStreamTest.java | 2 +-
.../util/FilePatternMatchingShardedFileTest.java | 2 +-
.../beam/sdk/util/MutationDetectorsTest.java | 6 +-
.../beam/sdk/util/NumberedShardedFileTest.java | 2 +-
.../beam/sdk/util/SerializableUtilsTest.java | 4 +-
.../apache/beam/sdk/util/WindowedValueTest.java | 4 +-
.../org/apache/beam/sdk/util/ZipFilesTest.java | 6 +-
.../java/org/apache/beam/sdk/values/KVTest.java | 2 +-
.../beam/sdk/values/PCollectionListTest.java | 2 +-
.../beam/sdk/values/PCollectionTupleTest.java | 2 +-
.../java/org/apache/beam/sdk/values/RowTest.java | 6 +-
.../org/apache/beam/sdk/values/TupleTagTest.java | 4 +-
.../apache/beam/sdk/values/TypeDescriptorTest.java | 2 +-
.../euphoria/core/client/operator/Distinct.java | 13 +-
.../euphoria/core/client/operator/Join.java | 13 +-
.../euphoria/core/client/operator/ReduceByKey.java | 13 +-
.../core/client/operator/ReduceWindow.java | 2 +-
.../euphoria/core/client/operator/Union.java | 2 +-
.../core/client/operator/WindowBuilder.java | 2 +-
.../core/client/util/PCollectionLists.java | 2 +-
.../core/translate/AbstractJoinTranslator.java | 11 +-
.../translate/BroadcastHashJoinTranslator.java | 61 ++-
.../translate/CompositeOperatorTranslator.java | 2 +-
.../euphoria/core/translate/JoinTranslator.java | 12 +-
.../euphoria/core/translate/OperatorTransform.java | 2 +-
.../core/translate/ReduceByKeyTranslator.java | 2 +-
.../provider/GenericTranslatorProvider.java | 2 +-
.../core/docs/DocumentationExamplesTest.java | 60 ++-
.../euphoria/core/testkit/ReduceByKeyTest.java | 2 +-
.../euphoria/core/testkit/WindowingTest.java | 73 +--
.../core/testkit/accumulators/NanosecondTimer.java | 2 +-
.../translate/BroadcastHashJoinTranslatorTest.java | 106 ++++
.../google-cloud-platform-core/build.gradle | 2 +-
.../sdk/extensions/gcp/options/GcpOptions.java | 10 +-
.../gcp/options/GcpPipelineOptionsRegistrar.java | 2 +-
.../sdk/extensions/gcp/options/GcsOptions.java | 14 +-
.../sdk/extensions/gcp/storage/GcsFileSystem.java | 36 +-
.../gcp/storage/GcsFileSystemRegistrar.java | 4 +-
.../extensions/gcp/storage/GcsPathValidator.java | 2 +-
.../sdk/extensions/gcp/storage/GcsResourceId.java | 6 +-
.../java/org/apache/beam/sdk/util/GcsUtil.java | 12 +-
.../java/org/apache/beam/sdk/util/Transport.java | 2 +-
.../org/apache/beam/sdk/util/gcsfs/GcsPath.java | 4 +-
.../sdk/extensions/gcp/GcpCoreApiSurfaceTest.java | 2 +-
.../sdk/extensions/gcp/options/GcpOptionsTest.java | 4 +-
.../gcp/storage/GcsFileSystemRegistrarTest.java | 2 +-
.../extensions/gcp/storage/GcsFileSystemTest.java | 4 +-
.../extensions/gcp/storage/GcsResourceIdTest.java | 29 +-
.../java/org/apache/beam/sdk/util/GcsUtilTest.java | 4 +-
sdks/java/extensions/jackson/build.gradle | 2 +-
.../beam/sdk/extensions/jackson/AsJsons.java | 2 +-
.../beam/sdk/extensions/jackson/ParseJsons.java | 2 +-
.../extensions/jackson/JacksonTransformsTest.java | 2 +-
sdks/java/extensions/join-library/build.gradle | 2 +-
.../beam/sdk/extensions/joinlibrary/Join.java | 2 +-
.../sdk/extensions/kryo/KryoCoderProvider.java | 2 +-
sdks/java/extensions/protobuf/build.gradle | 2 +-
.../sdk/extensions/protobuf/ByteStringCoder.java | 2 +-
.../beam/sdk/extensions/protobuf/ProtoCoder.java | 6 +-
.../protobuf/ProtobufCoderProviderRegistrar.java | 2 +-
.../beam/sdk/extensions/protobuf/ProtobufUtil.java | 2 +-
.../extensions/protobuf/ByteStringCoderTest.java | 2 +-
.../sdk/extensions/protobuf/ProtoCoderTest.java | 2 +-
.../sdk/extensions/protobuf/ProtobufUtilTest.java | 4 +-
sdks/java/extensions/sketching/build.gradle | 2 +-
.../extensions/sketching/ApproximateDistinct.java | 2 +-
.../extensions/sketching/SketchFrequencies.java | 2 +-
sdks/java/extensions/sorter/build.gradle | 2 +-
.../extensions/sorter/BufferedExternalSorter.java | 2 +-
.../beam/sdk/extensions/sorter/ExternalSorter.java | 4 +-
.../beam/sdk/extensions/sorter/InMemorySorter.java | 6 +-
.../sdk/extensions/sorter/SorterTestUtils.java | 2 +-
sdks/java/extensions/sql/build.gradle | 3 +-
.../sdk/extensions/sql/jdbc/BeamSqlLineIT.java | 4 +-
.../beam/sdk/extensions/sql/SqlTransform.java | 4 +-
.../sdk/extensions/sql/example/model/Customer.java | 2 +-
.../sdk/extensions/sql/example/model/Order.java | 2 +-
.../sdk/extensions/sql/impl/BeamCalciteSchema.java | 28 +-
.../sql/impl/BeamCalciteSchemaFactory.java | 180 ++++++-
.../sdk/extensions/sql/impl/BeamCalciteTable.java | 2 +-
.../sdk/extensions/sql/impl/BeamQueryPlanner.java | 7 +-
.../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 15 +-
.../sql/impl/CalciteConnectionWrapper.java | 400 +++++++++++++++
.../sdk/extensions/sql/impl/JdbcConnection.java | 128 +++++
.../beam/sdk/extensions/sql/impl/JdbcDriver.java | 149 +++---
.../extensions/sql/impl/ScalarFunctionImpl.java | 2 +-
.../sql/impl/UdfImplReflectiveFunctionBase.java | 2 +-
.../sql/impl/parser/SqlColumnDeclaration.java | 2 +-
.../sql/impl/parser/SqlCreateExternalTable.java | 2 +-
.../extensions/sql/impl/parser/SqlDropObject.java | 2 +-
.../sql/impl/parser/SqlSetOptionBeam.java | 21 +-
.../extensions/sql/impl/planner/BeamRuleSets.java | 2 +-
.../sql/impl/rel/BeamAggregationRel.java | 4 +-
.../sdk/extensions/sql/impl/rel/BeamCalcRel.java | 4 +-
.../sql/impl/rel/BeamEnumerableConverter.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamIOSinkRel.java | 2 +-
.../extensions/sql/impl/rel/BeamIOSourceRel.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamJoinRel.java | 4 +-
.../sql/impl/rel/BeamSetOperatorRelBase.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamSortRel.java | 2 +-
.../extensions/sql/impl/rel/BeamUncollectRel.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamUnnestRel.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamValuesRel.java | 4 +-
.../extensions/sql/impl/schema/BeamTableUtils.java | 6 +
.../impl/transform/BeamBuiltinAggregations.java | 2 +-
.../impl/transform/BeamSetOperatorsTransforms.java | 28 +-
.../sql/impl/transform/agg/CovarianceFn.java | 2 +-
.../sql/impl/utils/BigDecimalConverter.java | 2 +-
.../extensions/sql/impl/utils/CalciteUtils.java | 6 +-
.../sql/meta/provider/ReadOnlyTableProvider.java | 2 +-
.../sql/meta/provider/TableProvider.java | 5 +
.../sql/meta/provider/kafka/BeamKafkaTable.java | 2 +-
.../sql/meta/provider/test/TestTableProvider.java | 2 +-
.../sql/meta/provider/test/TestTableUtils.java | 2 +-
.../sql/meta/provider/text/TextTableProvider.java | 8 +-
.../sql/meta/store/InMemoryMetaStore.java | 2 +-
.../sdk/extensions/sql/BeamComplexTypeTest.java | 2 +-
.../sql/BeamSqlDslSqlStdOperatorsTest.java | 8 +-
.../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java | 2 +-
.../beam/sdk/extensions/sql/BeamSqlMapTest.java | 2 +-
.../extensions/sql/InferredJavaBeanSqlTest.java | 2 +-
.../sdk/extensions/sql/PubsubToBigqueryIT.java | 4 +-
.../apache/beam/sdk/extensions/sql/TestUtils.java | 2 +-
.../sdk/extensions/sql/impl/JdbcDriverTest.java | 2 +-
.../sql/impl/rel/BeamIntersectRelTest.java | 13 +-
.../extensions/sql/impl/rel/BeamMinusRelTest.java | 15 +-
...BeamSqlBuiltinFunctionsIntegrationTestBase.java | 6 +-
.../provider/bigquery/BigQueryReadWriteIT.java | 2 +-
.../provider/kafka/KafkaTableProviderTest.java | 2 +-
.../sql/meta/provider/pubsub/PubsubJsonIT.java | 25 +-
.../provider/pubsub/PubsubMessageToRowTest.java | 6 +-
.../meta/provider/text/TextTableProviderTest.java | 2 +-
.../beam/sdk/extensions/sql/utils/RowAsserts.java | 2 +-
sdks/java/fn-execution/build.gradle | 4 +-
.../beam/sdk/fn/channel/SocketAddressFactory.java | 4 +-
.../data/BeamFnDataBufferingOutboundObserver.java | 2 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 6 +-
.../beam/sdk/fn/data/RemoteGrpcPortRead.java | 4 +-
.../beam/sdk/fn/data/RemoteGrpcPortWrite.java | 4 +-
.../sdk/fn/stream/BufferingStreamObserver.java | 4 +-
.../org/apache/beam/sdk/fn/stream/DataStreams.java | 6 +-
.../org/apache/beam/sdk/fn/test/TestExecutors.java | 2 +-
.../sdk/fn/windowing/EncodedBoundedWindow.java | 2 +-
.../BeamFnDataBufferingOutboundObserverTest.java | 2 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 2 +-
.../sdk/fn/stream/BufferingStreamObserverTest.java | 4 +-
.../apache/beam/sdk/fn/stream/DataStreamsTest.java | 8 +-
.../sdk/fn/stream/DirectStreamObserverTest.java | 4 +-
sdks/java/harness/build.gradle | 6 +-
.../beam/fn/harness/AssignWindowsRunner.java | 8 +-
.../beam/fn/harness/BeamFnDataReadRunner.java | 6 +-
.../beam/fn/harness/BeamFnDataWriteRunner.java | 6 +-
.../beam/fn/harness/BoundedSourceRunner.java | 6 +-
.../org/apache/beam/fn/harness/CombineRunners.java | 8 +-
.../fn/harness/DoFnPTransformRunnerFactory.java | 14 +-
.../org/apache/beam/fn/harness/FlattenRunner.java | 8 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 10 +-
.../java/org/apache/beam/fn/harness/FnHarness.java | 2 +-
.../org/apache/beam/fn/harness/MapFnRunners.java | 6 +-
.../beam/fn/harness/PTransformRunnerFactory.java | 2 +-
.../beam/fn/harness/PrecombineGroupingTable.java | 6 +-
.../harness/SplittableProcessElementsRunner.java | 8 +-
.../beam/fn/harness/WindowMappingFnRunner.java | 2 +-
.../beam/fn/harness/WindowMergingFnRunner.java | 4 +-
.../harness/control/AddHarnessIdInterceptor.java | 2 +-
.../fn/harness/control/BeamFnControlClient.java | 4 +-
.../fn/harness/control/ProcessBundleHandler.java | 18 +-
.../harness/data/MultiplexingFnDataReceiver.java | 2 +-
.../fn/harness/logging/BeamFnLoggingClient.java | 6 +-
.../apache/beam/fn/harness/state/BagUserState.java | 4 +-
.../beam/fn/harness/state/FnApiStateAccessor.java | 8 +-
.../state/LazyCachingIteratorToIterable.java | 2 +-
.../fn/harness/state/StateFetchingIterators.java | 2 +-
.../beam/fn/harness/AssignWindowsRunnerTest.java | 10 +-
.../beam/fn/harness/BeamFnDataReadRunnerTest.java | 14 +-
.../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 10 +-
.../beam/fn/harness/BoundedSourceRunnerTest.java | 10 +-
.../apache/beam/fn/harness/CombineRunnersTest.java | 6 +-
.../apache/beam/fn/harness/FlattenRunnerTest.java | 8 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 10 +-
.../org/apache/beam/fn/harness/FnHarnessTest.java | 2 +-
.../apache/beam/fn/harness/MapFnRunnersTest.java | 10 +-
.../beam/fn/harness/WindowMergingFnRunnerTest.java | 6 +-
.../harness/control/BeamFnControlClientTest.java | 4 +-
.../harness/control/ProcessBundleHandlerTest.java | 6 +-
.../data/MultiplexingFnDataReceiverTest.java | 2 +-
.../harness/logging/BeamFnLoggingClientTest.java | 2 +-
.../beam/fn/harness/state/BagUserStateTest.java | 4 +-
.../state/BeamFnStateGrpcClientCacheTest.java | 2 +-
.../state/LazyCachingIteratorToIterableTest.java | 4 +-
.../fn/harness/state/MultimapSideInputTest.java | 4 +-
.../harness/state/StateFetchingIteratorsTest.java | 2 +-
sdks/java/io/amazon-web-services/build.gradle | 2 +-
.../apache/beam/sdk/io/aws/options/AwsModule.java | 2 +-
.../aws/options/AwsPipelineOptionsRegistrar.java | 2 +-
.../io/aws/s3/DefaultS3ClientBuilderFactory.java | 2 +-
.../apache/beam/sdk/io/aws/s3/S3FileSystem.java | 26 +-
.../beam/sdk/io/aws/s3/S3FileSystemRegistrar.java | 4 +-
.../io/aws/s3/S3ReadableSeekableByteChannel.java | 4 +-
.../apache/beam/sdk/io/aws/s3/S3ResourceId.java | 10 +-
.../beam/sdk/io/aws/s3/S3WritableByteChannel.java | 6 +-
.../beam/sdk/io/aws/sns/BasicSnsProvider.java | 2 +-
.../beam/sdk/io/aws/sns/PublishResultCoder.java | 2 +-
.../sdk/io/aws/sns/SnsCoderProviderRegistrar.java | 2 +-
.../java/org/apache/beam/sdk/io/aws/sns/SnsIO.java | 6 +-
.../beam/sdk/io/aws/sqs/SqsCheckpointMark.java | 4 +-
.../java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java | 2 +-
.../beam/sdk/io/aws/sqs/SqsUnboundedSource.java | 4 +-
.../beam/sdk/io/aws/s3/MatchResultMatcher.java | 6 +-
.../beam/sdk/io/aws/s3/S3FileSystemTest.java | 4 +-
.../org/apache/beam/sdk/io/aws/sns/SnsIOTest.java | 2 +-
sdks/java/io/amqp/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/amqp/AmqpIO.java | 4 +-
.../apache/beam/sdk/io/amqp/AmqpMessageCoder.java | 2 +-
.../io/amqp/AmqpMessageCoderProviderRegistrar.java | 2 +-
.../beam/sdk/io/amqp/AmqpMessageCoderTest.java | 2 +-
sdks/java/io/cassandra/build.gradle | 2 +-
.../apache/beam/sdk/io/cassandra/CassandraIO.java | 6 +-
.../sdk/io/cassandra/CassandraServiceImpl.java | 4 +-
.../beam/sdk/io/cassandra/CassandraIOTest.java | 2 +-
sdks/java/io/clickhouse/OWNERS | 4 +
sdks/java/io/{rabbitmq => clickhouse}/build.gradle | 42 +-
.../beam/sdk/io/clickhouse/ClickHouseIO.java | 516 ++++++++++++++++++++
.../beam/sdk/io/clickhouse/ClickHouseWriter.java | 134 +++++
.../apache/beam/sdk/io/clickhouse/TableSchema.java | 297 ++++++++++++
.../beam/sdk/io/clickhouse/package-info.java} | 21 +-
.../clickhouse/src/main/javacc/ColumnTypeParser.jj | 190 ++++++++
.../beam/sdk/io/clickhouse/AtomicInsertTest.java | 168 +++++++
.../beam/sdk/io/clickhouse/BaseClickHouseTest.java | 100 ++++
.../beam/sdk/io/clickhouse/ClickHouseIOTest.java | 361 ++++++++++++++
.../beam/sdk/io/clickhouse/TableSchemaTest.java | 144 ++++++
.../test/resources/config.d/zookeeper_default.xml | 23 +
sdks/java/io/common/build.gradle | 2 +-
.../org/apache/beam/sdk/io/common/HashingFn.java | 6 +-
.../org/apache/beam/sdk/io/common/TestRow.java | 2 +-
.../elasticsearch-tests-2/build.gradle | 2 +-
sdks/java/io/elasticsearch/build.gradle | 2 +-
.../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 6 +-
sdks/java/io/file-based-io-tests/build.gradle | 1 +
.../beam/sdk/io/common/FileBasedIOITHelper.java | 4 +-
.../io/common/FileBasedIOTestPipelineOptions.java | 19 +
.../java/org/apache/beam/sdk/io/text/TextIOIT.java | 44 +-
.../java/org/apache/beam/sdk/io/xml/XmlIOIT.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 10 +-
.../sdk/io/gcp/bigquery/BigQueryAvroUtils.java | 14 +-
.../bigquery/BigQueryCoderProviderRegistrar.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 8 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 18 +-
.../sdk/io/gcp/bigquery/BigQueryQuerySource.java | 4 +-
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 6 +-
.../sdk/io/gcp/bigquery/BigQuerySourceBase.java | 12 +-
.../sdk/io/gcp/bigquery/BigQueryTableSource.java | 8 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 4 +-
.../beam/sdk/io/gcp/bigquery/CreateTables.java | 10 +-
.../sdk/io/gcp/bigquery/DynamicDestinations.java | 4 +-
.../gcp/bigquery/DynamicDestinationsHelpers.java | 6 +-
.../sdk/io/gcp/bigquery/InsertRetryPolicy.java | 2 +-
.../io/gcp/bigquery/PassThroughThenCleanup.java | 2 +-
.../beam/sdk/io/gcp/bigquery/PrepareWrite.java | 2 +-
.../sdk/io/gcp/bigquery/StandardSQLTypeName.java | 4 +-
.../beam/sdk/io/gcp/bigquery/StreamingWriteFn.java | 4 +-
.../sdk/io/gcp/bigquery/TableRowInfoCoder.java | 2 +-
.../beam/sdk/io/gcp/bigquery/TableRowWriter.java | 4 +-
.../beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java | 2 +-
.../beam/sdk/io/gcp/bigquery/TestBigQuery.java | 2 +-
.../sdk/io/gcp/bigquery/WriteBundlesToFiles.java | 6 +-
.../beam/sdk/io/gcp/bigquery/WritePartition.java | 4 +-
.../beam/sdk/io/gcp/bigquery/WriteRename.java | 6 +-
.../beam/sdk/io/gcp/bigquery/WriteResult.java | 4 +-
.../beam/sdk/io/gcp/bigquery/WriteTables.java | 10 +-
.../beam/sdk/io/gcp/bigtable/BigtableConfig.java | 6 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 16 +-
.../sdk/io/gcp/bigtable/BigtableServiceImpl.java | 6 +-
.../gcp/common/GcpIoPipelineOptionsRegistrar.java | 2 +-
.../sdk/io/gcp/datastore/AdaptiveThrottler.java | 2 +-
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 16 +-
.../beam/sdk/io/gcp/pubsub/PubsubClient.java | 10 +-
.../gcp/pubsub/PubsubCoderProviderRegistrar.java | 2 +-
.../beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java | 8 +-
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 8 +-
.../beam/sdk/io/gcp/pubsub/PubsubJsonClient.java | 8 +-
.../beam/sdk/io/gcp/pubsub/PubsubMessage.java | 2 +-
.../gcp/pubsub/PubsubMessagePayloadOnlyCoder.java | 2 +-
.../beam/sdk/io/gcp/pubsub/PubsubTestClient.java | 6 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSink.java | 6 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSource.java | 12 +-
.../beam/sdk/io/gcp/pubsub/TestPubsubSignal.java | 8 +-
.../beam/sdk/io/gcp/spanner/BatchSpannerRead.java | 2 +-
.../sdk/io/gcp/spanner/MutationCellCounter.java | 2 +-
.../beam/sdk/io/gcp/spanner/MutationGroup.java | 4 +-
.../beam/sdk/io/gcp/spanner/MutationUtils.java | 2 +-
.../beam/sdk/io/gcp/spanner/NaiveSpannerRead.java | 2 +-
.../beam/sdk/io/gcp/spanner/OrderedCode.java | 10 +-
.../beam/sdk/io/gcp/spanner/SpannerConfig.java | 4 +-
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 12 +-
.../beam/sdk/io/gcp/spanner/SpannerSchema.java | 12 +-
.../sdk/io/gcp/spanner/SpannerWriteResult.java | 2 +-
.../beam/sdk/io/gcp/testing/BigqueryClient.java | 154 ++++--
.../beam/sdk/io/gcp/testing/BigqueryMatcher.java | 10 +-
.../cloud/spanner/FakeBatchTransactionId.java | 2 +-
.../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java | 2 +-
.../sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java | 17 +-
.../sdk/io/gcp/bigquery/BigQueryHelpersTest.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIOReadIT.java | 4 +-
.../sdk/io/gcp/bigquery/BigQueryIOReadTest.java | 6 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 20 +-
.../io/gcp/bigquery/BigQueryServicesImplTest.java | 6 +-
.../sdk/io/gcp/bigquery/BigQueryToTableIT.java | 199 ++++----
.../sdk/io/gcp/bigquery/InsertRetryPolicyTest.java | 2 +-
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 14 +-
.../beam/sdk/io/gcp/bigtable/BigtableWriteIT.java | 2 +-
.../beam/sdk/io/gcp/pubsub/PubsubClientTest.java | 2 +-
.../sdk/io/gcp/pubsub/PubsubGrpcClientTest.java | 6 +-
.../beam/sdk/io/gcp/pubsub/PubsubIOTest.java | 4 +-
.../sdk/io/gcp/pubsub/PubsubJsonClientTest.java | 4 +-
.../beam/sdk/io/gcp/pubsub/PubsubReadIT.java | 2 +-
.../sdk/io/gcp/pubsub/PubsubTestClientTest.java | 6 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSinkTest.java | 6 +-
.../io/gcp/pubsub/PubsubUnboundedSourceTest.java | 2 +-
.../sdk/io/gcp/spanner/MutationKeyEncoderTest.java | 2 +-
.../beam/sdk/io/gcp/spanner/OrderedCodeTest.java | 10 +-
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 6 +-
.../beam/sdk/io/gcp/spanner/SpannerWriteIT.java | 6 +-
.../sdk/io/gcp/testing/BigqueryMatcherTest.java | 2 +-
sdks/java/io/hadoop-file-system/build.gradle | 2 +-
.../apache/beam/sdk/io/hdfs/HadoopFileSystem.java | 4 +-
.../beam/sdk/io/hdfs/HadoopFileSystemOptions.java | 8 +-
.../io/hdfs/HadoopFileSystemOptionsRegistrar.java | 2 +-
.../sdk/io/hdfs/HadoopFileSystemRegistrar.java | 4 +-
.../apache/beam/sdk/io/hdfs/HadoopResourceId.java | 2 +-
.../hdfs/HadoopFileSystemOptionsRegistrarTest.java | 2 +-
.../sdk/io/hdfs/HadoopFileSystemOptionsTest.java | 6 +-
.../sdk/io/hdfs/HadoopFileSystemRegistrarTest.java | 6 +-
.../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 6 +-
sdks/java/io/hadoop-format/build.gradle | 2 +-
.../sdk/io/hadoop/format/HDFSSynchronization.java | 2 +-
.../beam/sdk/io/hadoop/format/HadoopFormatIO.java | 10 +-
.../beam/sdk/io/hadoop/format/HadoopFormats.java | 2 +-
.../sdk/io/hadoop/format/EmployeeInputFormat.java | 2 +-
.../hadoop/format/HadoopFormatIOElasticTest.java | 2 +-
.../format/ReuseObjectsEmployeeInputFormat.java | 2 +-
.../sdk/io/hadoop/format/TestEmployeeDataSet.java | 2 +-
sdks/java/io/hadoop-input-format/build.gradle | 2 +-
.../io/hadoop/inputformat/HadoopInputFormatIO.java | 2 +-
.../io/hadoop/inputformat/EmployeeInputFormat.java | 2 +-
.../hadoop/inputformat/HIFIOWithElasticTest.java | 2 +-
.../ReuseObjectsEmployeeInputFormat.java | 2 +-
.../io/hadoop/inputformat/TestEmployeeDataSet.java | 2 +-
sdks/java/io/hbase/build.gradle | 2 +-
.../sdk/io/hbase/HBaseCoderProviderRegistrar.java | 2 +-
.../java/org/apache/beam/sdk/io/hbase/HBaseIO.java | 4 +-
sdks/java/io/hcatalog/build.gradle | 2 +-
.../apache/beam/sdk/io/hcatalog/HCatalogIO.java | 4 +-
.../apache/beam/sdk/io/hcatalog/HCatalogIOIT.java | 2 +-
sdks/java/io/jdbc/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 29 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 2 +-
sdks/java/io/jms/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 4 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOTest.java | 11 +-
sdks/java/io/kafka/build.gradle | 2 +-
.../org/apache/beam/sdk/io/kafka/ConsumerSpEL.java | 6 +-
.../CustomTimestampPolicyWithLimitedDelay.java | 2 +-
.../beam/sdk/io/kafka/KafkaCheckpointMark.java | 2 +-
.../beam/sdk/io/kafka/KafkaExactlyOnceSink.java | 26 +-
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 20 +-
.../org/apache/beam/sdk/io/kafka/KafkaRecord.java | 2 +-
.../apache/beam/sdk/io/kafka/KafkaRecordCoder.java | 2 +-
.../beam/sdk/io/kafka/KafkaUnboundedReader.java | 22 +-
.../beam/sdk/io/kafka/KafkaUnboundedSource.java | 6 +-
.../org/apache/beam/sdk/io/kafka/KafkaWriter.java | 16 +-
.../org/apache/beam/sdk/io/kafka/ProducerSpEL.java | 2 +-
.../beam/sdk/io/kafka/TimestampPolicyFactory.java | 2 +-
.../CustomTimestampPolicyWithLimitedDelayTest.java | 5 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 132 ++++-
.../beam/sdk/io/kinesis/BasicKinesisProvider.java | 2 +-
.../sdk/io/kinesis/DynamicCheckpointGenerator.java | 5 +-
.../sdk/io/kinesis/GetKinesisRecordsResult.java | 3 +-
.../org/apache/beam/sdk/io/kinesis/KinesisIO.java | 2 +-
.../apache/beam/sdk/io/kinesis/KinesisReader.java | 2 +-
.../sdk/io/kinesis/KinesisReaderCheckpoint.java | 6 +-
.../apache/beam/sdk/io/kinesis/KinesisSource.java | 4 +-
.../apache/beam/sdk/io/kinesis/RecordFilter.java | 2 +-
.../beam/sdk/io/kinesis/ShardCheckpoint.java | 4 +-
.../beam/sdk/io/kinesis/ShardReadersPool.java | 10 +-
.../beam/sdk/io/kinesis/ShardRecordsIterator.java | 2 +-
.../sdk/io/kinesis/SimplifiedKinesisClient.java | 4 +-
.../apache/beam/sdk/io/kinesis/StartingPoint.java | 2 +-
.../sdk/io/kinesis/StartingPointShardsFinder.java | 2 +-
.../sdk/io/kinesis/StaticCheckpointGenerator.java | 2 +-
.../beam/sdk/io/kinesis/AmazonKinesisMock.java | 7 +-
.../io/kinesis/DynamicCheckpointGeneratorTest.java | 2 +-
.../apache/beam/sdk/io/kinesis/KinesisIOIT.java | 2 +-
.../beam/sdk/io/kinesis/KinesisMockReadTest.java | 4 +-
.../beam/sdk/io/kinesis/KinesisMockWriteTest.java | 4 +-
.../io/kinesis/KinesisReaderCheckpointTest.java | 2 +-
.../beam/sdk/io/kinesis/KinesisServiceMock.java | 2 +-
.../beam/sdk/io/kinesis/KinesisUploader.java | 2 +-
.../beam/sdk/io/kinesis/RecordFilterTest.java | 2 +-
.../beam/sdk/io/kinesis/ShardReadersPoolTest.java | 4 +-
.../io/kinesis/StartingPointShardsFinderTest.java | 2 +-
sdks/java/io/kudu/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/kudu/KuduIO.java | 12 +-
.../apache/beam/sdk/io/kudu/KuduServiceImpl.java | 4 +-
.../org/apache/beam/sdk/io/kudu/KuduTestUtils.java | 2 +-
sdks/java/io/mongodb/build.gradle | 2 +-
.../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 4 +-
.../org/apache/beam/sdk/io/mongodb/MongoDbIO.java | 4 +-
.../org/apache/beam/sdk/io/mongodb/SSLUtils.java | 2 +-
.../apache/beam/sdk/io/mongodb/MongoDBIOIT.java | 2 +-
sdks/java/io/mqtt/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/mqtt/MqttIO.java | 4 +-
sdks/java/io/parquet/build.gradle | 2 +-
.../org/apache/beam/sdk/io/parquet/ParquetIO.java | 2 +-
sdks/java/io/rabbitmq/build.gradle | 2 +-
.../apache/beam/sdk/io/rabbitmq/RabbitMqIO.java | 2 +-
.../beam/sdk/io/rabbitmq/RabbitMqIOTest.java | 12 +-
sdks/java/io/redis/build.gradle | 2 +-
.../sdk/io/redis/RedisConnectionConfiguration.java | 2 +-
.../java/org/apache/beam/sdk/io/redis/RedisIO.java | 6 +-
sdks/java/io/solr/build.gradle | 2 +-
.../beam/sdk/io/solr/AuthorizedSolrClient.java | 2 +-
.../java/org/apache/beam/sdk/io/solr/SolrIO.java | 10 +-
.../org/apache/beam/sdk/io/solr/SolrIOTest.java | 2 +-
sdks/java/io/synthetic/build.gradle | 2 +-
.../beam/sdk/io/synthetic/BundleSplitter.java | 67 +++
.../beam/sdk/io/synthetic/SyntheticBoundedIO.java | 436 -----------------
.../sdk/io/synthetic/SyntheticBoundedSource.java | 260 ++++++++++
.../beam/sdk/io/synthetic/SyntheticOptions.java | 8 +-
.../io/synthetic/SyntheticRecordsCheckpoint.java | 75 +++
.../sdk/io/synthetic/SyntheticSourceOptions.java | 174 +++++++
.../beam/sdk/io/synthetic/SyntheticStep.java | 15 +-
.../sdk/io/synthetic/SyntheticUnboundedSource.java | 202 ++++++++
.../beam/sdk/io/synthetic/SyntheticWatermark.java | 83 ++++
.../beam/sdk/io/synthetic/delay/ReaderDelay.java | 50 ++
.../SyntheticDelay.java} | 65 +--
.../beam/sdk/io/synthetic/delay/package-info.java} | 17 +-
.../beam/sdk/io/synthetic/BundleSplitterTest.java | 99 ++++
...IOTest.java => SyntheticBoundedSourceTest.java} | 17 +-
.../beam/sdk/io/synthetic/SyntheticStepTest.java | 2 +-
.../io/synthetic/SyntheticUnboundedSourceTest.java | 122 +++++
sdks/java/io/tika/build.gradle | 4 +-
.../org/apache/beam/sdk/io/tika/ParseResult.java | 8 +-
.../java/org/apache/beam/sdk/io/tika/TikaIO.java | 6 +-
sdks/java/io/xml/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/xml/JAXBCoder.java | 2 +-
.../java/org/apache/beam/sdk/io/xml/XmlIO.java | 4 +-
.../org/apache/beam/sdk/io/xml/JAXBCoderTest.java | 2 +-
.../java/org/apache/beam/sdk/io/xml/XmlIOTest.java | 2 +-
.../org/apache/beam/sdk/io/xml/XmlSourceTest.java | 2 +-
sdks/java/maven-archetypes/examples/build.gradle | 1 +
.../src/main/resources/archetype-resources/pom.xml | 27 +-
sdks/java/testing/load-tests/build.gradle | 30 +-
.../beam/sdk/loadtests/CoGroupByKeyLoadTest.java | 20 +-
.../apache/beam/sdk/loadtests/CombineLoadTest.java | 23 +-
.../beam/sdk/loadtests/GroupByKeyLoadTest.java | 23 +-
.../org/apache/beam/sdk/loadtests/LoadTest.java | 25 +-
.../apache/beam/sdk/loadtests/LoadTestOptions.java | 3 +-
.../apache/beam/sdk/loadtests/LoadTestResult.java | 2 +-
.../apache/beam/sdk/loadtests/ParDoLoadTest.java | 16 +-
.../loadtests/SyntheticDataPubSubPublisher.java | 18 +-
sdks/java/testing/nexmark/build.gradle | 2 +-
.../java/org/apache/beam/sdk/nexmark/Main.java | 4 +-
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 12 +-
.../org/apache/beam/sdk/nexmark/NexmarkUtils.java | 14 +-
.../org/apache/beam/sdk/nexmark/PubsubHelper.java | 8 +-
.../org/apache/beam/sdk/nexmark/model/Auction.java | 4 +-
.../apache/beam/sdk/nexmark/model/AuctionBid.java | 2 +-
.../beam/sdk/nexmark/model/AuctionCount.java | 2 +-
.../beam/sdk/nexmark/model/AuctionPrice.java | 2 +-
.../org/apache/beam/sdk/nexmark/model/Bid.java | 2 +-
.../beam/sdk/nexmark/model/BidsPerSession.java | 2 +-
.../beam/sdk/nexmark/model/CategoryPrice.java | 2 +-
.../org/apache/beam/sdk/nexmark/model/Done.java | 2 +-
.../org/apache/beam/sdk/nexmark/model/Event.java | 4 +-
.../beam/sdk/nexmark/model/IdNameReserve.java | 2 +-
.../beam/sdk/nexmark/model/NameCityStateId.java | 2 +-
.../org/apache/beam/sdk/nexmark/model/Person.java | 4 +-
.../apache/beam/sdk/nexmark/model/SellerPrice.java | 2 +-
.../sdk/nexmark/queries/BoundedSideInputJoin.java | 2 +-
.../apache/beam/sdk/nexmark/queries/Query10.java | 2 +-
.../apache/beam/sdk/nexmark/queries/Query3.java | 2 +-
.../beam/sdk/nexmark/queries/Query3Model.java | 4 +-
.../apache/beam/sdk/nexmark/queries/Query6.java | 2 +-
.../beam/sdk/nexmark/queries/Query8Model.java | 4 +-
.../sdk/nexmark/queries/SessionSideInputJoin.java | 2 +-
.../nexmark/queries/SessionSideInputJoinModel.java | 12 +-
.../beam/sdk/nexmark/queries/WinningBids.java | 2 +-
.../queries/sql/SqlBoundedSideInputJoin.java | 2 +-
.../beam/sdk/nexmark/queries/sql/SqlQuery5.java | 2 +-
.../sdk/nexmark/sources/generator/Generator.java | 2 +-
.../sources/generator/GeneratorCheckpoint.java | 2 +-
.../beam/sdk/nexmark/model/sql/RowSizeTest.java | 2 +-
.../nexmark/queries/BoundedSideInputJoinTest.java | 2 +-
.../nexmark/queries/SessionSideInputJoinTest.java | 2 +-
.../queries/sql/SqlBoundedSideInputJoinTest.java | 2 +-
.../sdk/nexmark/queries/sql/SqlQuery2Test.java | 2 +-
.../sdk/nexmark/queries/sql/SqlQuery3Test.java | 2 +-
.../sdk/nexmark/queries/sql/SqlQuery5Test.java | 2 +-
.../sdk/nexmark/queries/sql/SqlQuery7Test.java | 2 +-
sdks/java/testing/test-utils/build.gradle | 2 +-
.../apache/beam/sdk/testutils/NamedTestResult.java | 76 +++
.../beam/sdk/testutils/metrics/MetricsReader.java | 6 +-
.../sdk/testutils/publishing/BigQueryClient.java | 4 +-
.../publishing/BigQueryResultsPublisher.java | 5 +-
.../publishing/BigQueryResultsPublisherTest.java | 2 +-
sdks/python/apache_beam/coders/coders.py | 9 +-
.../apache_beam/examples/complete/tfidf_test.py | 2 +-
.../cookbook/datastore_wordcount_it_test.py | 2 +-
.../examples/wordcount_debugging_test.py | 4 +-
.../apache_beam/examples/wordcount_it_test.py | 2 +-
.../apache_beam/examples/wordcount_minimal_test.py | 4 +-
sdks/python/apache_beam/examples/wordcount_test.py | 4 +-
sdks/python/apache_beam/io/filebasedsource_test.py | 47 +-
sdks/python/apache_beam/io/gcp/__init__.py | 22 +
.../apache_beam/io/gcp/datastore/v1/datastoreio.py | 6 +-
.../apache_beam/io/gcp/datastore/v1/helper.py | 12 +-
sdks/python/apache_beam/io/gcp/gcsio_test.py | 19 +
sdks/python/apache_beam/io/range_trackers.py | 28 +-
sdks/python/apache_beam/io/range_trackers_test.py | 133 ++---
.../apache_beam/io/source_test_utils_test.py | 29 +-
sdks/python/apache_beam/io/sources_test.py | 18 +-
sdks/python/apache_beam/io/textio.py | 2 +-
sdks/python/apache_beam/io/textio_test.py | 203 ++++----
sdks/python/apache_beam/pipeline_test.py | 1 +
sdks/python/apache_beam/runners/common.py | 2 -
.../runners/interactive/cache_manager.py | 3 +-
.../runners/interactive/cache_manager_test.py | 4 +-
.../runners/interactive/interactive_runner_test.py | 8 -
.../runners/interactive/pipeline_analyzer_test.py | 11 -
.../runners/portability/flink_runner_test.py | 7 +-
.../runners/portability/fn_api_runner_test.py | 21 -
.../runners/portability/portable_runner.py | 13 +-
.../apache_beam/runners/worker/opcounters_test.py | 7 +-
.../runners/worker/statesampler_test.py | 32 +-
sdks/python/apache_beam/testing/util.py | 2 +-
sdks/python/apache_beam/transforms/combiners.py | 36 +-
.../apache_beam/transforms/combiners_test.py | 26 +-
sdks/python/apache_beam/transforms/core.py | 40 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/build.gradle | 11 +-
sdks/python/setup.py | 5 +-
sdks/python/tox.ini | 2 +-
settings.gradle | 2 +
vendor/grpc-1_13_1/build.gradle | 4 +-
website/Gemfile.lock | 16 +-
website/Rakefile | 13 +-
website/src/_data/capability-matrix.yml | 114 +++++
website/src/_includes/section-menu/community.html | 1 +
.../src/_includes/section-menu/documentation.html | 34 +-
website/src/_includes/section-menu/roadmap.html | 1 +
website/src/_includes/section-menu/runners.html | 1 +
website/src/_includes/section-menu/sdks.html | 1 -
website/src/_posts/2017-01-09-added-apex-runner.md | 4 +-
website/src/_posts/2017-08-04-splittable-do-fn.md | 2 +-
...2018-08-20-review-input-streaming-connectors.md | 2 +-
website/src/community/integrations.md | 36 ++
website/src/contribute/index.md | 2 +-
website/src/documentation/index.md | 3 +-
website/src/documentation/io/authoring-java.md | 37 --
website/src/documentation/io/authoring-overview.md | 103 ----
website/src/documentation/io/authoring-python.md | 32 --
.../documentation/io/built-in-google-bigquery.md | 2 +-
website/src/documentation/io/built-in-hadoop.md | 2 -
website/src/documentation/io/built-in-hcatalog.md | 2 -
website/src/documentation/io/built-in.md | 2 -
website/src/documentation/io/contributing.md | 29 --
website/src/documentation/io/developing-io-java.md | 368 ++++++++++++++
.../src/documentation/io/developing-io-overview.md | 170 +++++++
.../src/documentation/io/developing-io-python.md | 281 +++++++++++
website/src/documentation/io/io-toc.md | 40 --
website/src/documentation/io/testing.md | 3 -
website/src/documentation/programming-guide.md | 48 +-
.../documentation/resources/learning-resources.md | 125 +++++
.../videos-and-podcasts.md} | 34 +-
website/src/documentation/runners/apex.md | 2 +-
website/src/documentation/runners/nemo.md | 169 +++++++
website/src/documentation/sdks/euphoria.md | 4 +-
website/src/documentation/sdks/python-custom-io.md | 307 ------------
website/src/documentation/sdks/python.md | 8 +-
website/src/get-started/beam-overview.md | 4 +-
website/src/get-started/mobile-gaming-example.md | 7 +
website/src/get-started/quickstart-go.md | 8 +-
website/src/get-started/quickstart-java.md | 38 +-
website/src/get-started/quickstart-py.md | 8 +-
website/src/get-started/wordcount-example.md | 71 +++
website/src/roadmap/nemo-runner.md | 27 ++
1789 files changed, 16207 insertions(+), 6851 deletions(-)
copy sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/PCollectionLists.java => .test-infra/jenkins/CommonTestProperties.groovy (58%)
create mode 100644 .test-infra/jenkins/LoadTestsBuilder.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_Java.groovy
copy runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/{SerializablePipelineOptions.java => PipelineOptionsSerializationUtils.java} (58%)
rename runners/{samza/src/main/java/org/apache/beam/runners/samza/util => core-java/src/main/java/org/apache/beam/runners/core/serialization}/Base64Serializer.java (94%)
copy sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/PCollectionLists.java => runners/core-java/src/main/java/org/apache/beam/runners/core/serialization/package-info.java (63%)
create mode 100644 runners/spark/src/test/java/org/apache/beam/runners/spark/coders/BeamSparkRunnerRegistratorTest.java
create mode 100644 runners/spark/src/test/java/org/apache/beam/runners/spark/translation/GroupCombineFunctionsTest.java
create mode 100644 sdks/go/pkg/beam/beam.shims.go
create mode 100644 sdks/go/pkg/beam/core/runtime/coderx/coderx.shims.go
create mode 100644 sdks/go/pkg/beam/testing/passert/passert.shims.go
create mode 100644 sdks/go/pkg/beam/transforms/filter/filter.shims.go
create mode 100644 sdks/go/pkg/beam/transforms/top/top.shims.go
create mode 100644 sdks/go/pkg/beam/x/debug/debug.shims.go
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsOptions.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/AutoValueSchema.java
rename sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/{ => annotations}/DefaultSchema.java (90%)
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/annotations/SchemaCreate.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/annotations/SchemaFieldName.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/annotations/SchemaIgnore.java
copy sdks/java/{extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/PCollectionLists.java => core/src/main/java/org/apache/beam/sdk/schemas/annotations/package-info.java} (63%)
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AutoValueUtils.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/AutoValueSchemaTest.java
create mode 100644 sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/translate/BroadcastHashJoinTranslatorTest.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/CalciteConnectionWrapper.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcConnection.java
create mode 100644 sdks/java/io/clickhouse/OWNERS
copy sdks/java/io/{rabbitmq => clickhouse}/build.gradle (53%)
create mode 100644 sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIO.java
create mode 100644 sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/ClickHouseWriter.java
create mode 100644 sdks/java/io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/TableSchema.java
copy sdks/java/{extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/PCollectionLists.java => io/clickhouse/src/main/java/org/apache/beam/sdk/io/clickhouse/package-info.java} (63%)
create mode 100644 sdks/java/io/clickhouse/src/main/javacc/ColumnTypeParser.jj
create mode 100644 sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/AtomicInsertTest.java
create mode 100644 sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/BaseClickHouseTest.java
create mode 100644 sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/ClickHouseIOTest.java
create mode 100644 sdks/java/io/clickhouse/src/test/java/org/apache/beam/sdk/io/clickhouse/TableSchemaTest.java
create mode 100644 sdks/java/io/clickhouse/src/test/resources/config.d/zookeeper_default.xml
create mode 100644 sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/BundleSplitter.java
delete mode 100644 sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticBoundedIO.java
create mode 100644 sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticBoundedSource.java
create mode 100644 sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticRecordsCheckpoint.java
create mode 100644 sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticSourceOptions.java
create mode 100644 sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticUnboundedSource.java
create mode 100644 sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticWatermark.java
create mode 100644 sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/delay/ReaderDelay.java
rename sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/{SyntheticUtils.java => delay/SyntheticDelay.java} (90%)
copy sdks/java/{extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/PCollectionLists.java => io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/delay/package-info.java} (63%)
create mode 100644 sdks/java/io/synthetic/src/test/java/org/apache/beam/sdk/io/synthetic/BundleSplitterTest.java
rename sdks/java/io/synthetic/src/test/java/org/apache/beam/sdk/io/synthetic/{SyntheticBoundedIOTest.java => SyntheticBoundedSourceTest.java} (93%)
create mode 100644 sdks/java/io/synthetic/src/test/java/org/apache/beam/sdk/io/synthetic/SyntheticUnboundedSourceTest.java
create mode 100644 sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/NamedTestResult.java
create mode 100644 website/src/community/integrations.md
delete mode 100644 website/src/documentation/io/authoring-java.md
delete mode 100644 website/src/documentation/io/authoring-overview.md
delete mode 100644 website/src/documentation/io/authoring-python.md
delete mode 100644 website/src/documentation/io/contributing.md
create mode 100644 website/src/documentation/io/developing-io-java.md
create mode 100644 website/src/documentation/io/developing-io-overview.md
create mode 100644 website/src/documentation/io/developing-io-python.md
delete mode 100644 website/src/documentation/io/io-toc.md
create mode 100644 website/src/documentation/resources/learning-resources.md
rename website/src/documentation/{resources.md => resources/videos-and-podcasts.md} (78%)
create mode 100644 website/src/documentation/runners/nemo.md
delete mode 100644 website/src/documentation/sdks/python-custom-io.md
create mode 100644 website/src/roadmap/nemo-runner.md
[beam] 01/01: Prepare for 2.10.0 release
Posted by ke...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch release-2.10.0
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 343e6629b0830509a1123484ac766207fdac4328
Author: Kenneth Knowles <ke...@apache.org>
AuthorDate: Tue Jan 15 12:08:00 2019 -0800
Prepare for 2.10.0 release
---
runners/google-cloud-dataflow-java/build.gradle | 2 +-
sdks/python/apache_beam/version.py | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/runners/google-cloud-dataflow-java/build.gradle b/runners/google-cloud-dataflow-java/build.gradle
index c4c721f..f913502 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -39,7 +39,7 @@ processResources {
filter org.apache.tools.ant.filters.ReplaceTokens, tokens: [
'dataflow.legacy_environment_major_version' : '7',
'dataflow.fnapi_environment_major_version' : '7',
- 'dataflow.container_version' : 'beam-master-20190110'
+ 'dataflow.container_version' : 'beam-2.10.0'
]
}
diff --git a/sdks/python/apache_beam/version.py b/sdks/python/apache_beam/version.py
index 57512fe..4e5e679 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -18,4 +18,4 @@
"""Apache Beam SDK version information and utilities."""
-__version__ = '2.10.0.dev'
+__version__ = '2.10.0'